systemd Unit Files & journalctl

Reading unit files, writing simple ones, controlling services, and finding log output with journalctl.

systemd basics

systemd is the init system and service manager on all modern RHEL/Debian-based distributions. It starts and stops services, manages boot ordering, handles system targets (similar to old runlevels), and collects logs via journald.

Everything systemd manages is a unit. Services are .service units. There are also .timer, .socket, .target, and .mount units. When you say "start the nginx service", you are telling systemd to activate the nginx.service unit.

Essential systemctl commands

# Start / stop / restart a service
systemctl start nginx
systemctl stop nginx
systemctl restart nginx

# Reload config without full restart (if the service supports it)
systemctl reload nginx

# Enable at boot / disable at boot
systemctl enable nginx
systemctl disable nginx

# Enable AND start in one command
systemctl enable --now nginx
systemctl disable --now nginx

# Check status
systemctl status nginx

# Is it running?
systemctl is-active nginx        # prints: active or inactive

# Is it enabled at boot?
systemctl is-enabled nginx       # prints: enabled or disabled

# List all running services
systemctl list-units --type=service --state=running

# List failed services
systemctl --failed

# Clear the "failed" state after you have investigated and fixed
systemctl reset-failed nginx
systemctl reset-failed     # clear all failed units

# Mask a unit — prevents it from being started at all (even manually)
# Use when a package installs a service you never want running
systemctl mask postfix
# To re-enable it:
systemctl unmask postfix

# disable stops it starting at boot; mask makes it impossible to start
# disable: "don't start on boot"   mask: "block completely"

# View the unit file
systemctl cat nginx

Unit file anatomy

A service unit file has three sections: [Unit], [Service], and [Install].

[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Key directives explained:

• After= — start this service after these units (ordering, not requirement)
• Requires= — if this service fails, stop mine too (hard dependency)
• Wants= — start those units too, but continue even if they fail (soft dependency)
• Type= — how systemd knows the service is started:
  • simple — the ExecStart process IS the main process (most common)
  • forking — the process forks and the parent exits; systemd tracks the child via PIDFile
  • oneshot — runs once and exits; systemd waits for it to finish
  • notify — service tells systemd when it is ready via sd_notify
• ExecStart= — the command to run to start the service
• ExecStartPre= — commands to run before ExecStart (e.g. config validation)
• Restart= — when to automatically restart:
  • on-failure — restart if the process exits with a non-zero code
  • always — restart regardless of how it exited
  • no — never restart (useful for one-shot tasks)
• WantedBy=multi-user.target — enable this service in normal (non-graphical) boot

Resource limits in unit files

[Service]
# Limit CPU to 50% of one core
CPUQuota=50%

# Limit RAM to 512 MB (kills the process if exceeded)
MemoryMax=512M
MemoryHigh=400M   # soft limit — systemd starts throttling before killing

# Limit open file descriptors (common for databases and web servers)
LimitNOFILE=65536

# Limit number of processes this service can create
LimitNPROC=512

Resource limits are useful for multi-tenant servers and for preventing runaway services from taking down the whole host. Check current limits for a running service with systemctl show nginx | grep -i limit. Changes take effect after a service restart.

Where unit files live

/lib/systemd/system/        # package-installed units (do not edit these)
/usr/lib/systemd/system/    # same, on RHEL
/etc/systemd/system/        # your overrides and custom units (edit here)
~/.config/systemd/user/     # user-level units (for non-root services)

Files in /etc/systemd/system/ take precedence over files with the same name in /lib/systemd/system/. This is how you override a package-provided unit without modifying it.

Writing a simple service unit

Creating a systemd service for a custom application:

# /etc/systemd/system/myapp.service

[Unit]
Description=My Application
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yml
Restart=on-failure
RestartSec=5s

# Resource limits
LimitNOFILE=65536

# Security hardening (optional but recommended)
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# After creating the file, reload systemd and start the service
systemctl daemon-reload
systemctl enable --now myapp
systemctl status myapp

Override without editing the original

To change a setting in a package-provided unit file without editing the original (which would be overwritten on package update):

# The easy way — opens the correct override file in an editor
systemctl edit nginx

# This creates /etc/systemd/system/nginx.service.d/override.conf
# Add ONLY the directives you want to change:
[Service]
Restart=always
RestartSec=10s
Environment="EXTRA_OPTS=-p /var/run/nginx_custom.pid"

systemctl daemon-reload
systemctl restart nginx

To see the effective unit file after overrides are applied:

systemctl cat nginx

The override file only needs to contain the sections and directives you are changing. systemd merges it with the original. You do not need to copy the entire unit file.

Service dependencies and ordering

[Unit]
# Start after these (ordering only — they can fail)
After=network.target rsyslog.service

# Require these to be running (if they fail, stop me too)
Requires=postgresql.service

# Start those alongside me if possible (soft want)
Wants=optional-helper.service

# Tightly couples this unit's lifecycle to another — if the bound
# unit stops or fails, this unit stops too (stronger than Requires)
BindsTo=required-service.service

journalctl — reading logs

journalctl reads the systemd journal — the central log store that captures output from all systemd services.

# View logs for a specific service
journalctl -u nginx

# Follow logs in real time (like tail -f)
journalctl -u nginx -f

# Show last 50 lines
journalctl -u nginx -n 50

# Show logs since last boot
journalctl -u nginx -b

# Show logs from previous boot (when a service crashed at boot)
journalctl -u nginx -b -1

# Show errors and above only
journalctl -u nginx -p err

# Show all kernel messages (useful for OOM / hardware issues)
journalctl -k

# Show all logs since a specific time
journalctl --since "2024-10-01 08:00:00"
journalctl --since "1 hour ago"

Useful journalctl flags

-u nginx            # filter by unit name
-f                  # follow (live)
-n 100              # last 100 lines
-b                  # current boot
-b -1               # previous boot
-p err              # errors and above (emerg, alert, crit, err)
-p warning          # warnings and above
--since "1h ago"    # time filter
--until "10 min ago"
--no-pager          # don't page output (useful in scripts)
-o short            # short format (default)
-o json             # JSON format
-o cat              # just the message, no metadata

# Show all failed service logs from the current boot
journalctl -b -p err --no-pager

# Combine filters
journalctl -u nginx -b --since "30 min ago" -p warning

Making logs persistent

By default on some systems, the journal does not persist across reboots (it lives in /run/log/journal/ which is tmpfs).

# Check where journal is stored
ls /var/log/journal/    # exists = persistent
ls /run/log/journal/    # exists only = volatile

# Enable persistence
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

# Or set in config
echo "Storage=persistent" >> /etc/systemd/journald.conf
systemctl restart systemd-journald

# Control how much disk space the journal uses
# In /etc/systemd/journald.conf:
SystemMaxUse=2G       # max disk space
SystemKeepFree=500M   # always keep this free
MaxRetentionSec=30d   # delete logs older than 30 days

# Vacuum old logs manually
journalctl --vacuum-size=2G
journalctl --vacuum-time=30d

timedatectl — time and timezone

timedatectl is the systemd tool for checking and configuring the system clock and timezone. It replaces the older date and hwclock workflows on systemd-based systems.

# Check current time, timezone, and NTP sync status
timedatectl
# Output shows: Local time, Universal time, RTC time, Time zone, NTP active, Synchronized

# Set timezone
timedatectl set-timezone Europe/London
timedatectl set-timezone America/New_York
timedatectl set-timezone UTC

# List available timezones (pipe through grep)
timedatectl list-timezones
timedatectl list-timezones | grep Australia

# Check if NTP is synchronized
timedatectl show --property=NTPSynchronized
timedatectl show --property=NTPService

# Enable NTP sync (uses systemd-timesyncd or chrony, whichever is configured)
timedatectl set-ntp true

If Chrony is installed, it takes over NTP duties and timedatectl will report its sync status. See the Chrony page for diagnosing time sync issues. On RHEL, chronyd is the default; on Debian/Ubuntu, systemd-timesyncd is the lightweight default but Chrony is preferred for servers.

loginctl — sessions and users

loginctl is the systemd-logind interface for inspecting and managing interactive sessions. Useful on multi-user hosts, jump boxes, and any time you need to answer "who's logged in and how?"

# Who is currently logged in, on which sessions
loginctl list-sessions
# SESSION  UID USER   SEAT  TTY
#       3 1001 alice        pts/0
#       5 1002 bob          pts/1

# Details of one session (idle time, type, service, leader PID)
loginctl session-status 3

# Users known to logind (including lingering enabled)
loginctl list-users

# Terminate a session or all sessions for a user
loginctl terminate-session 3
loginctl terminate-user bob

# Enable user-level services to run even when the user is logged out
loginctl enable-linger alice

enable-linger is the switch that makes per-user systemctl --user services survive logout — required for rootless Podman containers or user timers that need to keep running on a headless machine.

Transient units (systemd-run)

A transient unit is a service, scope, or timer created on the fly — no unit file on disk, no reload, and it disappears after it exits (or at the next boot). Use it when you want systemd's supervision, logging, or resource limits for a one-off command without the ceremony of authoring a real unit.

# Run a long-running job as a named service — output goes to the journal
systemd-run --unit=backup-now --description="Ad-hoc backup" \
  /usr/local/bin/backup.sh /srv /backup

# Follow its output
journalctl -u backup-now -f

# Run under a cgroup scope so ps / htop show it grouped
systemd-run --scope --unit=mytool bash

# Apply resource limits to an interactive command
systemd-run --scope -p CPUQuota=20% -p MemoryMax=512M stress --cpu 4

# Schedule a one-shot timer ("run in 10 minutes")
systemd-run --on-active=10m --unit=cleanup-later /usr/local/bin/cleanup.sh

systemd-cgls — the cgroup tree

systemd-cgls prints the cgroup hierarchy as a tree, so you can see at a glance which services own which processes — invaluable when a runaway process shows up in top and you want to know the unit it belongs to.

systemd-cgls --no-pager

Sample output:

Control group /:
-.slice
├─user.slice
│ └─user-1000.slice
│   ├─user@1000.service
│   │ └─init.scope
│   │   └─1284 /lib/systemd/systemd --user
│   └─session-3.scope
│     ├─1290 sshd: alice [priv]
│     └─1312 -bash
├─system.slice
│ ├─nginx.service
│ │ ├─ 842 "nginx: master process"
│ │ ├─ 843 "nginx: worker"
│ │ └─ 844 "nginx: worker"
│ ├─postgresql.service
│ │ └─ 910 /usr/lib/postgresql/16/bin/postgres
│ └─cron.service
│   └─ 611 /usr/sbin/cron -f
└─init.scope
  └─1 /sbin/init

Pair with systemd-cgtop for a live view sorted by CPU or memory — it's top but grouped by cgroup/unit instead of by PID, which is far more useful for answering "which service is eating my RAM?"