Rsyslog Appears to Hang When Queues Fill¶

When an output action is slower than the incoming log rate, rsyslog queues can fill. A full main or ruleset queue can make inputs slow down while they wait for space, which can look like rsyslog or local applications are hanging.

Why this happens¶

Rsyslog uses queues to decouple inputs, rulesets, and actions. If a destination such as a remote server, database, or helper program is unavailable, the affected queue may grow until it reaches its configured limits. At that point rsyslog applies flow control instead of accepting unlimited messages.

For the main message queue and ruleset queues, this can affect inputs. Local applications that write to a blocking log socket may then also slow down because rsyslog is no longer draining that socket quickly. This is usually not a deadlock inside rsyslog; it is backpressure caused by a downstream queue that cannot make progress.

The key setting is queue.timeoutEnqueue. When a queue is full, rsyslog waits up to that timeout for space to become available. The default avoids immediately dropping messages, but a large value on the main queue can make the whole system feel stuck under sustained pressure. A value of 0 discards immediately when the queue cannot accept a message.

Confirm the condition¶

Enable impstats and inspect the queue counters while the problem is happening:

size near queue.size shows the queue is full or nearly full.
full increases when the queue cannot accept more messages.
discarded.full increases when messages are dropped because the queue is full.
discarded.nf increases when the discard watermark is active.
Action counters with suspended or repeated failures identify the slow or unavailable destination.

Also check rsyslog’s own diagnostics for messages about suspended actions, queue full conditions, disk queue problems, or failed output destinations.

Reduce the blast radius¶

Prefer isolating slow destinations behind dedicated action queues. For example, a remote forwarding action can have its own disk-assisted queue so local file outputs and unrelated actions continue while the remote target is offline:

global(workDirectory="/var/spool/rsyslog")

action(
    type="omfwd"
    target="loghost.example.net"
    protocol="tcp"
    queue.type="LinkedList"
    queue.filename="fwd"
    queue.maxDiskSpace="2g"
    queue.saveOnShutdown="on"
    action.resumeRetryCount="-1"
)

For the main queue, avoid very large queue.timeoutEnqueue values unless the host is intentionally allowed to slow log producers instead of losing messages. Choose an explicit policy for overload:

buffer temporarily with enough disk space and monitoring;
discard lower-priority messages with queue.discardMark and queue.discardSeverity;
discard immediately on a full queue with queue.timeoutEnqueue="0";
increase queue size only when memory and disk capacity are understood.

Operational checklist¶

When diagnosing this symptom:

verify the unavailable or slow action first;
enable impstats and alert on queue size, full, and discard counters;
use per-action queues for remote, database, or helper-program outputs;
set queue.maxDiskSpace on disk-assisted queues so spool files cannot fill the filesystem silently;
decide up front whether the system should prefer blocking, buffering, or dropping during sustained overload.

Rsyslog Appears to Hang When Queues Fill¶

Why this happens¶

Confirm the condition¶

Reduce the blast radius¶

Operational checklist¶

See also¶