systemd Unit Authoring

Writing correct, hardened unit files: Type= choice, exec hooks, Restart policies, dependency ordering, drop-ins, timers, socket activation, and user units.

Unit file anatomy

A service unit has three sections. [Unit] describes the unit and its dependencies. [Service] describes how to run the process. [Install] tells systemctl enable where to hook the unit.

[Unit]
Description=Example app
Documentation=https://example.com/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/example --config /etc/example.conf
User=example
Group=example
Restart=on-failure
RestartSec=3s

[Install]
WantedBy=multi-user.target

Canonical paths:

Type= values

Picking the wrong type is the most common cause of systemctl start "succeeding" while the service is actually broken.

Type=simple

Default. The process specified by ExecStart is the main process and runs in the foreground. systemd considers the unit "active" as soon as it forks the process — it does not wait for the process to be ready. Use when the program has no readiness signal.

Type=exec

Like simple, but systemd waits until execve() has completed before considering the unit started. Prefer over simple when available (systemd ≥ 240).

Type=notify

The daemon calls sd_notify(READY=1) when initialisation is complete. systemd only marks the unit active after that message. This is what you want for anything with a real readiness signal (nginx in notify mode, a long-booting Java app after it opens its port).

Type=forking

The program double-forks and exits; the real daemon keeps running. systemd tracks the grandchild via PIDFile=. Legacy pattern — avoid writing new daemons this way.

Type=oneshot

Runs one or more ExecStart= lines to completion, then exits. By default the unit is considered inactive once finished; add RemainAfterExit=yes to keep it "active" (useful for setup jobs whose effect persists).

Type=idle

Like simple, but execution is delayed until active jobs are dispatched. Used for getty so the console is quiet during boot.

Type=dbus

Unit is considered ready when the named bus name appears on the system bus (BusName=).

ExecStart, ExecStartPre, ExecStartPost, ExecStop

[Service]
Type=simple
ExecStartPre=/usr/bin/install -d -o app -g app -m 0750 /var/lib/app
ExecStartPre=-/usr/bin/rm -f /run/app/app.sock
ExecStart=/usr/local/bin/app --config /etc/app.conf
ExecStartPost=/usr/local/bin/healthcheck.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
TimeoutStartSec=30s
TimeoutStopSec=15s

• A leading - on an Exec line (ExecStartPre=-/usr/bin/rm -f ...) means "ignore non-zero exit". Use for cleanups that may or may not find anything.
• A leading + runs with full privileges even if User= is set — sometimes needed for a pre-step that touches a root-owned path.
• ExecStart must be an absolute path. No shell metacharacters unless you launch /bin/sh -c explicitly.
• Use $MAINPID, $EXIT_STATUS, and $INVOCATION_ID in post/stop hooks.

Restart= policies and rate limits

[Service]
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60s
StartLimitBurst=5               # if 5 restarts inside 60s, stop trying

StartLimitIntervalSec= and StartLimitBurst= live in [Unit] on older systemd, in [Service] on newer. Without them, a crash-loop can pin a CPU re-executing a broken binary.

Dependency ordering: After/Wants/Requires/BindsTo

Ordering (After=, Before=) and activation (Wants=, Requires=, Requisite=, BindsTo=) are orthogonal. Most bugs are from confusing them.

After=foo.service

Order only. If foo is being started, finish it before me. Does not cause foo to start.

Wants=foo.service

When I am started, also start foo. If foo fails, I keep going. Most common coupling.

Requires=foo.service

When I am started, also start foo. If foo fails to start or is stopped later, I am stopped too. Strong.

Requisite=foo.service

Fail fast: if foo is not already active, fail immediately without starting it.

BindsTo=foo.service

Like Requires but also reacts to sudden termination of foo, not just clean stop.

PartOf=foo.service

When foo is restarted or stopped, do the same to me.

Conflicts=bar.service

Starting me stops bar, and vice versa.

[Unit]
Description=App
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

Environment and working dir

[Service]
WorkingDirectory=/var/lib/app
Environment="DB_URL=postgres://app@127.0.0.1/app"
Environment="LOG_LEVEL=info"
EnvironmentFile=/etc/app/app.env       # KEY=VALUE lines; '-' prefix = optional
EnvironmentFile=-/etc/app/app.local    # ignore if missing

PassEnvironment=HTTP_PROXY NO_PROXY    # inherit from systemd's environment

• EnvironmentFile= parses shell-style KEY=VALUE, but it is not a shell — no $FOO substitution, no export.
• Prefix with - to make the file optional. Useful for split secret/non-secret files.
• Avoid placing secrets in Environment= directly; they show up in systemctl show.

Hardening: ProtectSystem, PrivateTmp, caps, syscalls

Treat the following as a default baseline for any service that does not specifically need more privilege:

[Service]
User=app
Group=app
DynamicUser=no                  # set yes if you do not need a stable UID
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict            # /usr, /boot, /etc read-only
ProtectHome=yes                 # /home, /root, /run/user hidden
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
ReadWritePaths=/var/lib/app /var/log/app
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
UMask=0027

Journal integration

[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=app                   # tag
SyslogLevel=info
LogRateLimitIntervalSec=30s
LogRateLimitBurst=1000

• Default is journal, so most units need no explicit setting.
• Use StandardOutput=append:/var/log/app/app.log to write a file in addition (rare; prefer journald + journalctl --output=json pipelines).
• If a daemon emits massive log volume at startup, raise LogRateLimitBurst or disable it (LogRateLimitIntervalSec=0) to avoid dropped lines.

Drop-ins

Never edit a vendor unit in /usr/lib/systemd/system/. Create a drop-in that overrides or appends:

systemctl edit sshd.service            # creates /etc/systemd/system/sshd.service.d/override.conf
systemctl edit --full sshd.service     # full copy to /etc/systemd/system/ (prefer drop-in)
systemctl daemon-reload
systemctl restart sshd

# /etc/systemd/system/sshd.service.d/override.conf
[Service]
LimitNOFILE=65535
# To CLEAR an inherited list, assign empty first:
ExecStart=
ExecStart=/usr/sbin/sshd -D -f /etc/ssh/sshd_config.custom

Timers

Timers are units of type .timer that activate a matching .service. Prefer them over cron for anything that needs dependency handling, persistent catch-up, or tight journald integration.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/backup.timer
[Unit]
Description=Run nightly backup

[Timer]
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=10m
Persistent=true
AccuracySec=1min
Unit=backup.service

[Install]
WantedBy=timers.target

systemctl enable --now backup.timer
systemctl list-timers --all
systemd-analyze calendar "*-*-* 02:15:00" --iterations=3

• Persistent=true runs the job on next boot if it was missed while the host was off. Without it, missed runs are lost.
• RandomizedDelaySec= jitters the start — crucial on fleets to avoid stampeding NFS, IPA, or a CI runner pool.
• OnBootSec=5min + OnUnitActiveSec=1h is the idiom for "5 minutes after boot, then every hour".

Socket activation

systemd can listen on a port and only start the service when a connection arrives — useful for rarely-used services, inetd-style tools, and faster boot.

# /etc/systemd/system/echo.socket
[Unit]
Description=Echo on 2007

[Socket]
ListenStream=2007
Accept=no
NoDelay=true

[Install]
WantedBy=sockets.target

# /etc/systemd/system/echo.service
[Unit]
Requires=echo.socket
After=echo.socket

[Service]
ExecStart=/usr/local/bin/echod
StandardInput=socket

systemctl enable --now echo.socket
ss -ltnp | grep 2007
nc localhost 2007

With Accept=yes, systemd forks a template unit (echo@.service) per connection; with Accept=no the daemon inherits the listening socket via fd 3 and handles concurrency itself. Most modern daemons want the latter.

User units

Per-user services live under ~/.config/systemd/user/ and are managed with systemctl --user. They run inside a user manager started at login.

loginctl enable-linger alice            # allow alice's user manager to run without an active session

mkdir -p ~/.config/systemd/user/
$EDITOR ~/.config/systemd/user/mybot.service

systemctl --user daemon-reload
systemctl --user enable --now mybot
journalctl --user -u mybot -f

[Unit]
Description=User bot

[Service]
Type=simple
ExecStart=%h/bin/bot.py
Restart=on-failure

[Install]
WantedBy=default.target

Specifiers worth knowing: %h user home, %u user name, %U UID, %t runtime dir (/run/user/$UID), %n full unit name, %i instance name (for foo@bar.service).

Example 1 — simple notify service

# /etc/systemd/system/metricsd.service
[Unit]
Description=Metrics collector
Documentation=https://example.com/metricsd
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main
WatchdogSec=30s
ExecStart=/usr/local/bin/metricsd --config /etc/metricsd.yml
Restart=on-failure
RestartSec=3s
StartLimitIntervalSec=60s
StartLimitBurst=5

User=metricsd
Group=metricsd
DynamicUser=no

NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/metricsd
CapabilityBoundingSet=
AmbientCapabilities=
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Example 2 — hardened Python worker

# /etc/systemd/system/billing-worker@.service
[Unit]
Description=Billing worker %i
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service redis.service
PartOf=billing.target

[Service]
Type=simple
EnvironmentFile=/etc/billing/worker.env
EnvironmentFile=-/etc/billing/worker.local
WorkingDirectory=/opt/billing
ExecStart=/opt/billing/venv/bin/python -m billing.worker --queue %i
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
TimeoutStopSec=30s
Restart=always
RestartSec=5s

User=billing
Group=billing
UMask=0027

NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
ReadWritePaths=/var/lib/billing /var/log/billing
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources @mount @reboot
CapabilityBoundingSet=

LimitNOFILE=32768
LimitNPROC=512

[Install]
WantedBy=multi-user.target

systemctl enable --now billing-worker@invoices.service
systemctl enable --now billing-worker@statements.service
systemctl status 'billing-worker@*'

Example 3 — oneshot plus timer

# /etc/systemd/system/certbot-renew.service
[Unit]
Description=Renew Let's Encrypt certs
Documentation=https://certbot.eff.org/docs/using.html
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet --agree-tos --deploy-hook "systemctl reload nginx"
Nice=10
IOSchedulingClass=idle
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=yes
TimeoutStartSec=15min

# /etc/systemd/system/certbot-renew.timer
[Unit]
Description=Twice-daily certbot renewal

[Timer]
OnCalendar=*-*-* 03,15:12:00
RandomizedDelaySec=30m
Persistent=true
Unit=certbot-renew.service

[Install]
WantedBy=timers.target

systemd-analyze verify /etc/systemd/system/certbot-renew.*
systemctl enable --now certbot-renew.timer
systemctl list-timers certbot-renew.timer

Troubleshooting

Cross-reference

• systemd & journalctl — reading logs from units authored here.
• SELinux and SELinux debugging — many hardening options interact with SELinux domains.
• cron & Timers — when to keep cron, when to move to timers.
• Bash scripting — writing clean ExecStart wrappers.
• Service troubleshooting — runbook-style triage for dead services.