Rsyslog Basics

Page 11 — Local log handling and remote forwarding.

What Rsyslog is

Rsyslog handles logging. It can:

• Collect local logs from running services
• Write logs to files in /var/log/
• Forward logs to a centralised remote log server

Logs are often the first place you look when something breaks. Rsyslog is what collects and routes them.

Main config locations

/etc/rsyslog.conf
/etc/rsyslog.d/

The main config is in rsyslog.conf. Drop-in files go in rsyslog.d/ — useful for keeping custom forwarding rules separate.

Facilities and severities

Every log message has a facility (the source/category) and a severity (how serious it is).

Common facilities

authpriv · mail · daemon · kern · local0–local7

Severity levels (high to low)

emerg · alert · crit · err · warning · notice · info · debug

Forwarding example

# Forward everything via TCP
*.* @@logserver.example.com:514

# Forward everything via UDP
*.* @logserver.example.com:514

@@ (double at) = TCP. @ (single at) = UDP. TCP is more reliable. Port 514 is the standard syslog port.

The *.* selector means all facilities and all severities. You can be more specific:

authpriv.*  @@logserver.example.com:514
mail.err    @@logserver.example.com:514

Service checks

systemctl status rsyslog
systemctl restart rsyslog
journalctl -u rsyslog -n 50

What to check

• Service is running
• Config is valid (no errors on restart)
• Remote port reachable: nc -zv logserver 514
• Logs are arriving on the remote server
• SELinux or firewall is not blocking the port

Config validation (rsyslogd -N1)

Before reloading rsyslog after any config change, validate the config without actually starting the daemon. This prevents a bad config from breaking logging system-wide.

# Validate the full rsyslog config (level 1 = config check only)
rsyslogd -N1

# Validate and show all loaded config files
rsyslogd -N1 -f /etc/rsyslog.conf

# If valid, output ends with:
# rsyslogd: End of config validation run. Bye.

# If invalid, you get a specific error:
# rsyslogd: error: invalid input module 'imtcp': No such file or directory

Always run rsyslogd -N1 before systemctl restart rsyslog. A bad config silently stops all local logging, which you may only notice much later during an incident investigation.

logger smoke test

The logger command writes a syslog message from the command line — the fastest way to verify that log routing works end-to-end after a config change.

# Send a test message with a specific facility and severity
logger -p mail.info "Test message from logger"

# Then verify it appeared in the expected log file
grep "Test message" /var/log/maillog   # RHEL/CentOS
grep "Test message" /var/log/mail.log  # Debian/Ubuntu

# Send a test to verify remote forwarding (check on the remote server)
logger -p local0.info "Remote log routing test from $(hostname)"

# Custom tag (appears in log as "mytag: message")
logger -t mytag -p daemon.warning "Something happened on $(hostname)"

# Use logger to test a full facility → file routing rule
logger -p auth.info "SSH auth test message"
tail -5 /var/log/secure   # or /var/log/auth.log

Receiving logs (imtcp)

To set up rsyslog as a central log receiver (log aggregator), load the TCP input module and define a listener. All sending hosts forward to this server.

On the receiving server

# /etc/rsyslog.conf or /etc/rsyslog.d/receiver.conf

# Load the TCP input module
module(load="imtcp")

# Open the TCP listener on port 514
input(type="imtcp" port="514")

# Optional: also load UDP (legacy senders)
module(load="imudp")
input(type="imudp" port="514")

# Store logs per sending host in separate files
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~    # stop processing here (don't also log to local files)

# Open firewall port on the receiver
firewall-cmd --permanent --add-port=514/tcp
firewall-cmd --permanent --add-port=514/udp
firewall-cmd --reload

# Validate and restart
rsyslogd -N1
systemctl restart rsyslog

# Verify listener is open
ss -tlnp | grep :514

On each sending host

# /etc/rsyslog.d/forward.conf
# Forward all logs to the central syslog server via TCP
*.* @@logserver.internal.example.com:514   # @@ = TCP, @ = UDP

# Or with the newer RainerScript syntax
action(type="omfwd"
  target="logserver.internal.example.com"
  port="514"
  protocol="tcp"
  action.resumeRetryCount="10"
  queue.type="LinkedList"
  queue.size="10000")

The newer RainerScript syntax with queue settings is preferred — it buffers messages in memory if the central server is temporarily unreachable and retries automatically.

journald integration (imjournal)

On systemd hosts, most services log through journald rather than writing to /dev/log. The imjournal input module pulls messages straight out of the journal — complete with structured fields (_SYSTEMD_UNIT, _PID, _UID, PRIORITY) that imuxsock never sees.

# /etc/rsyslog.conf — load imjournal instead of (or in addition to) imuxsock
$ModLoad imjournal
$imjournalRatelimitInterval 600
$imjournalRatelimitBurst 20000

# Persistent cursor so rsyslog resumes at the last-read entry
# across restarts (no lost or duplicated messages).
$imjournalStateFile /var/lib/rsyslog/imjournal.state

Structured data from journald is preserved as message properties, so you can reference them in templates and filters:

# Route everything systemd logs for sshd into its own file,
# using the structured _SYSTEMD_UNIT field from journald.
if $!_SYSTEMD_UNIT == "sshd.service" then {
    action(type="omfile" file="/var/log/sshd-journal.log")
    stop
}

TLS forwarding

Plain-text 514/tcp is fine on a trusted management VLAN, but over any untrusted link you want TLS. Rsyslog ships two TLS drivers — gtls (GnuTLS) and ossl (OpenSSL) — plus the more robust RELP protocol for guaranteed delivery. For the full walk-through including mutual auth and RELP, see Rsyslog Forwarding.

Minimal imtcp + gnutls receiver

# /etc/rsyslog.d/tls-receiver.conf (on the central log server)

global(
  defaultNetstreamDriver="gtls"
  defaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.crt"
  defaultNetstreamDriverCertFile="/etc/pki/rsyslog/server.crt"
  defaultNetstreamDriverKeyFile="/etc/pki/rsyslog/server.key"
)

module(load="imtcp"
  StreamDriver.Name="gtls"
  StreamDriver.Mode="1"              # 1 = TLS required
  StreamDriver.AuthMode="x509/name") # verify client cert CN/SAN

input(type="imtcp" port="6514")      # IANA syslog-tls port

Matching sender

# /etc/rsyslog.d/tls-forward.conf (on each host)

global(
  defaultNetstreamDriver="gtls"
  defaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.crt"
  defaultNetstreamDriverCertFile="/etc/pki/rsyslog/client.crt"
  defaultNetstreamDriverKeyFile="/etc/pki/rsyslog/client.key"
)

action(type="omfwd"
  target="logserver.internal.example.com"
  port="6514"
  protocol="tcp"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverAuthMode="x509/name"
  StreamDriverPermittedPeers="logserver.internal.example.com"
  queue.type="LinkedList"
  queue.fileName="tls-fwd"
  queue.saveOnShutdown="on")

RELP (omrelp/imrelp, port 2514 or 20514) is preferred when zero message loss matters — unlike plain TCP it acknowledges each message at the application layer, so a dropped connection never silently drops log lines. RELP also supports TLS via the same gtls driver.