Forwarding syslog with rsyslog over TCP behind a disk-assisted queue

Goal

Forward all local log traffic from an rsyslog host to a central collector over TCP, with a local buffer that keeps messages queued on disk if the collector or the network is unavailable, instead of dropping them.

Prerequisites

Root access; rsyslog installed and running (rsyslogd -v); the collector's address and port; free disk space on a filesystem the rsyslog user can write to for the spool directory.

Steps

1. The distribution's default /etc/rsyslog.conf (Debian, Ubuntu, RHEL) includes every file in /etc/rsyslog.d/ matching *.conf, in file-name sort order; the documentation's own example uses a numbered file such as /etc/rsyslog.d/10-myapp.conf. Create a new drop-in rather than editing rsyslog.conf directly, so the change is isolated and easy to remove: /etc/rsyslog.d/60-forward.conf.
2. In that file, define a disk-assisted queue and an action that forwards everything over TCP:

# queue files go to the workDirectory already set in rsyslog.conf
action(type="omfwd"
       target="collector.example.org" port="514" protocol="tcp"
       queue.type="LinkedList"
       queue.filename="fwd01"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on"
       action.resumeRetryCount="-1")

queue.type="LinkedList" plus queue.filename makes this a disk-assisted queue: in memory normally, spilling to disk under the work directory once it reaches its high watermark (default 90% of queue.size, which defaults to 1000 messages for an action queue) and, with queue.saveOnShutdown="on", saving queued messages at shutdown. action.resumeRetryCount="-1" retries forwarding indefinitely instead of giving up after a fixed number of attempts. 3. Find the work directory with grep -ri workdirectory /etc/rsyslog.conf (Debian/Ubuntu: /var/spool/rsyslog; RHEL: /var/lib/rsyslog) and do not set it a second time. Ensure it is writable by the user rsyslogd runs as and has enough free space for the expected outage window; queue.maxDiskSpace caps how much it can consume. 4. Validate the configuration without restarting the running daemon: rsyslogd -N1 -f /etc/rsyslog.conf performs a config check only, per the -N option in rsyslogd(8); it prints errors and does not start logging. 5. Apply the change: systemctl restart rsyslog. A reload is not enough: per rsyslogd(8), SIGHUP only makes rsyslogd close its open files; it does not re-read the configuration. A restart is not a system reboot. 6. Test forwarding: logger "forward test $(date -u +%FT%TZ)" on the source host, then check the message arrives on the collector. Simulate an outage by blocking the collector port with a firewall rule, generate more messages than the high watermark (e.g. 2000 logger calls; a handful stays in memory), and confirm queue files appear in the work directory and drain once the block is removed.

Expected result

rsyslogd -N1 reports no errors; test messages reach the collector; during a simulated outage, queue files grow in the work directory and are sent once connectivity returns, with no gap in the collector's log beyond the outage window.

Limits and test basis

Based on rsyslog's documentation on include files, queues, and RainerScript queue parameters, and on rsyslogd(8). To undo, remove the drop-in file and restart rsyslog; queue files left in the work directory can be deleted once drained. A queue with unlimited retry and no queue.maxDiskSpace limit can fill the disk during a long outage — always set a cap sized to the partition.