Config Reload on HUP: Design Record¶
Design record for config reload on HUP. A HUP first validates the changed
config in a child rsyslogd -N1. Later it loads a never-activated donor
config in process and patches the running config at a per-batch gate in
msgConsumer(). Ruleset bodies come first, listeners and settings later.
Everything is off by default.
Status¶
Accepted design, implementation in progress. Tracking issue: #8007.
This page records requirements, invariants and decisions so that reviewers and later contributors can check each pull request against them. It is not user documentation; the user-facing setting is documented with the release that ships it. Function names are cited instead of line numbers because line numbers go stale.
Nothing described here is user visible yet. Until a milestone ships, a HUP does exactly what it did before: it runs the legacy HUP hooks (file reopen, lookup table reload, and similar) and nothing else.
Goal and requirements¶
End goal: a HUP applies configuration changes while established sessions, in particular TCP sessions, stay up, as far as the change permits.
Requirements for every step on the way:
No throughput loss. Ruleset execution is the hot path. The design must not add per-message locks, lookups, reference counts or atomics. The one accepted exception is direct-mode main and ruleset queues, where a batch is a single message, so the per-batch gate cost becomes a per-message cost (see the gate sketch below). Direct-mode main queues are documented as not recommended. From the milestone that touches the data path on, a benchmark against
mainis an acceptance criterion, and it must include a direct-mode configuration.Fail safe. A change that cannot be applied leaves the running config untouched and is reported with a reason that names the object and, where known, the source location. The operator then fixes the config or restarts.
No
exit()on any path reachable after startup.Both frontends. RainerScript and YAML (including YAML with
script:blocks) yield the same decisions. Every reload test has a YAML twin.Tests run in parallel (
make check -j) without relying on timing.Small, shippable steps. Each pull request delivers something usable or is a standalone fix. No slice may depend on a later slice to be correct.
OFF equals main. With the feature off, behaviour and output are the same as before the feature existed (see OFF-mode gate).
New source files carry an SPDX Apache-2.0 header.
User-visible setting¶
global(config.reloadOnHUP="off|report|on")
off(default): today’s HUP behaviour.report: on HUP, if the config files changed, validate them and log one outcome line. Nothing is applied.on: apply the supported part of a change; reject the whole change, with the reason, if any part is unsupported.
The legacy HUP hooks always run first and unchanged, in every mode. A reload
attempt only happens when the files read at load time changed.
report and on stay documented as experimental until live reload has
soaked in production use.
Invariants¶
These hold for every milestone. A pull request that needs to break one has to change this page first.
Single commit point. A reload either changes nothing or commits at exactly one point. Failure before commit leaves the running config as it was. Problems after commit are retirement errors: they are logged and counted, never undone by switching back.
The running
rsconf_tand everyruleset_tstay alive. They are patched in place.runConfis never replaced at runtime, so the many unsynchronised readers ofrunConfstay valid.The candidate config is a donor. It is loaded detached, compared with the running config, and the changed objects are moved out of it. It is never passed to
activate(), so the missing rollback inactivate()and the privileged pre-drop hooks are not reload problems.Messages hold a stable pointer.
smsg_tkeeps its directruleset_t *. Only the contents of the ruleset (its body) change, at a batch boundary.Swaps happen only at the gate. Ruleset code is entered in exactly one place,
msgConsumer()intools/rsyslogd.c, for main, ruleset, disk-assisted and direct-mode queues. A swap happens only while every thread is outside that function. Action queues never run ruleset code and are never gated.Candidate objects are invisible. A candidate must not register statistics, open outputs, bind sockets or start threads before it is committed. One exception: the queues of new actions are started, still empty, in the prepare step, because starting a queue can fail and must fail before the commit point. Their worker threads exist then, but receive no messages until the swap. If the reload fails, these queues are stopped and destroyed again. Their statistics objects are published only at commit, so
qqueueStart()needs a variant that defersstatsobj.ConstructFinalize().References resolve to running objects. When the candidate defines an object (template, lookup table, dynstats or percentile bucket, ruleset) equal to a running one, name lookups during the candidate load return the running object. An adopted tree therefore never points into the donor.
Unchanged objects keep their identity. Reused actions keep their object, queue, statistics and action number. Action numbers are never reused within one process lifetime.
Every change is classified. A change the current milestone does not support rejects the whole reload, with the reason. Nothing is applied partially.
Architecture¶
Reload runs on the main thread, in the main loop, after doHUP(). Shutdown
and classic HUP also run there, so they can delay a reload but never overlap
with it. A termination request aborts a pending reload.
Steps of one reload attempt (on mode, full form):
Change check. Compare the recorded config file list (files opened by the RainerScript lexer and the YAML loader, include globs and their expansions, with stat data and a content hash) against the files on disk. No change means no further work.
Pre-flight. Run the same binary as a child with the original config arguments and
-N1, with a timeout, without blocking the main loop. The child inherits the dropped privileges and the chroot, so it sees exactly what an in-process load can read. An invalid config stops here.Donor load. Load the candidate in process through the normal loader, which must be repeatable for this (milestone M2).
Classify each difference against the reload-class table. Anything out of scope rejects.
Prepare everything the swap needs: allocations, action renumbering, new action queues started (still empty, statistics not yet published; see invariant 6).
Gate. Request the gate and wait for every gated thread to be outside
msgConsumer(), bounded by a quiesce timeout.Swap. Pointer stores only, no allocation, no I/O. Cannot fail.
Release the gate.
Retire old bodies and the rest of the donor.
Report one log line and update counters and the generation number.
Gate sketch (milestone M3): a global word holds a “reload pending” bit and a
config epoch, written only by the main thread. Each wti_t publishes
whether it is inside msgConsumer(), with a nesting depth so that a nested
entry (call into a direct-mode ruleset queue) never parks. Entry and the
reloader’s request form a store-then-load pair with full barriers on both
sides. Per batch this costs a few stores to the worker’s own wti_t and one
load of a read-mostly word. In direct mode a batch is one message, so the
cost there is per message: one full barrier per message on that path. This
is the exception named under the requirements; the M3 benchmark measures it,
and if it is too expensive, direct-mode queues make on unavailable
instead. All wti_t objects are registered, so idle
workers and workers being started or stopped are covered. Worker cancellation
cleanup resets the gate state.
Worker action tables grow at the gate: each wti_t records the size of its
actWrkrInfo array, and a thread resizes its own table when it sees a new
epoch. Every scan of the table is bounded by that size, which is already the
case on main.
Decisions and rejected alternatives¶
Chosen:
Normal loader, used as a donor, over a private parse-and-materialise pipeline. The loader is streaming (
cnfDoObj()and action construction run during the parse), so a private pipeline has to re-implement object construction. That second implementation would be a permanent maintenance cost.Patch the running config in place over config generations or new handle types.
ruleset_talready is the pointer-stable shell.Child-process pre-flight first. It gives real validation with no loader changes and no data-path risk, and it stays useful as crash isolation in front of the in-process load.
Module opt-in for repeatable load through the existing
isCompatibleWithFeature()query. Modules without the flag make a config with them restart-only. No module interface version bump.Processing graph before listeners. Replacing ruleset bodies already lets TCP sessions survive the most common edits, because sessions route to
ruleset_tshells that stay. imtcp listener changes follow per instance; “stop and start the whole module” is not an acceptable intermediate step.Capture on top of
runtime/translate.cfor canonical per-object descriptions and the diff. Legacy$directives are captured verbatim and in order.
Rejected (see the closed pull requests listed in #8007 for the code; the
prototype stack is also recorded as config-reload-private-pipeline in
doc/ai/rejected_engineering_paths.md):
Stopping and starting inputs in process: loses TCP sessions.
Full
Load()plusactivate()of a new config on HUP:activate()switchesrunConfbefore several fallible steps without rollback, and privileged initialisation cannot be repeated after privilege drop.Listener file-descriptor hand-off or
SO_REUSEPORT: old sessions keep the old config.A private syntax parse with a separate normalised graph and materialiser (the 2026 Codex prototype stack): second object-construction path, large shared-path changes, OFF-mode regressions.
Two live versions of each ruleset body with per-batch selection (no parking): never stalls, but every other shared structure would need its own concurrent update, and the per-message path gains an indexed load. Revisit only if the bounded stall proves unacceptable.
Milestones¶
- M0 - groundwork (no behaviour change)
This record; config destructors act on the config they are given; worker action tables bounded by their own size; grammar
%destructorentries and lexer ownership so failed parses do not leak; exact HUP completion in the testbench (GetHUPCount/AwaitHUPCount); testbench helpers that report failures.- M1 -
reportmode File record and change check at startup and on HUP; pre-flight child with timeout, output capture and coalescing of HUP bursts; one outcome line (unchanged, invalid with first error, valid and changed); statistics counters. Ships as experimental.
- M2 - repeatable in-process load and classification
One-time initialisation separated from per-load work; legacy directives resolve through the config being loaded; side effects and statistics registration deferred to activation; failed candidates freed;
exit()in load paths turned into error returns (startup keeps today’s exit codes); repeatable-load module flag; reference resolution to running objects; canonical descriptions and diff;reportbecomes object-level.- M3 - swap primitives, inert when off
Per-batch gate,
wti_tregistry,callresolving through the ruleset instead of a cached body pointer, growable action tables, stable action identity and numbers, an imdiag command that swaps a body with an identical one in a loop. Benchmark againstmainis the exit criterion.- M4 - first live reload: ruleset bodies
M4a: changed bodies of existing rulesets (default ruleset included), unchanged actions reused, new actions with direct or in-memory queues, new templates. M4b: removed and changed actions, changed templates, added and removed rulesets (a removed ruleset’s shell routes to the default ruleset and counts).
- M5 - imtcp listeners, then other inputs
Unchanged instances and their sessions untouched; add, remove (with drain deadline) and rebind instances. Then a restartable flag for other inputs.
- M6 - base settings
A reviewed subset of
global()parameters, TLS certificate reload for new sessions, queue tunables, lookup, dynstats, percentile and parser-chain definitions.- M7 - widen and harden
Disk and disk-assisted queue changes, systemd
Type=notify-reload, generated reload-class documentation, soak testing.
Reload-class table¶
Every config object and setting gets exactly one class. The table is filled in as milestones land; until an entry says otherwise, the class is restart. Where this table and the restart-only list below disagree, the restart-only list wins.
Object or setting |
Class |
Notes |
|---|---|---|
Ruleset body (filters, statements) |
live (M4a) |
Swapped at the gate; sessions and queued messages keep their ruleset |
Unchanged action |
reused (M4a) |
Keeps object, queue, counters and number |
New action, direct or in-memory queue |
live (M4a) |
Constructed and started before commit |
New template |
live (M4a) |
|
Removed or changed action, changed template |
live (M4b) |
Drained with the configured shutdown timeouts; memory and direct queues only |
Changed action with a disk or disk-assisted queue |
restart (until M7) |
A new queue on the same spool file needs stop-old-before-start-new |
Added or removed ruleset |
live (M4b) |
Removed ruleset becomes a tombstone routing to the default ruleset |
imtcp instance added, removed or rebound |
live (M5) |
Other instances and their sessions untouched |
Selected |
live (M6) |
Per parameter, after review |
Everything else |
restart |
See the restart-only list |
Restart-only list¶
These stay restart-only until a milestone explicitly moves them:
privilege, capability, chroot and umask settings;
workDirectory,maxMessageSize,debug.*,environment,stdlog.channelspec,operatingStateFile;main queue type and file name;
ruleset queue parameters and parser chains (input-side parsing in
runtime/ratelimit.cruns outside the gate) until M6;changed actions with disk-backed queues (a new queue on the same spool file needs stop-old-before-start-new) until M7;
experimental local queues;
configs that use a module without the repeatable-load flag;
inputs without the restartable flag;
module unload;
anything given on the command line.
OFF-mode gate¶
Every pull request of this effort must show that off behaves like
main:
-N1output identical tomainover the test configs.Existing HUP tests unchanged and green.
From M3 on, a throughput comparison against
main.Shared-path changes are reviewed against the list of regressions that the earlier prototype introduced in OFF mode:
one listener bind failure stopping all imtcp listeners; changed tcpsrv shutdown semantics; per-connection
regcomp; periodicpoll()wake-ups;an unknown legacy property aborting the whole parse; optional include I/O errors becoming fatal; new error or info lines at every startup or HUP;
$ResetConfigVariablesdroppingglobal()ormain_queue()settings; lostallowedSenderdiagnostics; the whole config cloned and hashed even when off;SIGTERMremoved from the blocked signal set (lost wake-up); dequeue time window ignored at shutdown; extra mutex pairs per message.
Testing expectations¶
RainerScript and YAML twins for every reload test.
HUP tests wait with
issue_HUP(exact completion count), never with sleeps.Reload paths run under ASan, UBSan and TSan lanes; the M2 load/destroy loop and the M3 swap loop run under ASan and TSan.
Live reload tests (M4 on) check message conservation under load, repeated reloads with file descriptor, thread and memory checks, invalid candidates under load, suspended actions, reload racing HUP and shutdown, and an established imtcp session that survives and uses the new body.
Open points¶
Recorded defaults apply until decided otherwise:
Quiesce timeout: 2 s with up to 3 retries before a reload is reported as not applied.
Direct-mode main queues: supported through the same gate; decided after the TSan lane exists.
Performance budget for M3: within run-to-run noise and at most 1 % on the imtcp-to-omfile benchmark, measured on quiet hardware.
Platforms:
reportwherever spawning a child works;ondeveloped and gated on Linux, refused with a clear message where untested.
Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project
Contributing: Source & docs: rsyslog source project
© 2008–2026 Rainer Gerhards and others. Licensed under the Apache License 2.0.