impstats: Generate Periodic Statistics of Internal Counters¶

Module Name:	impstats
Author:	Rainer Gerhards <rgerhards@adiscon.com>

Purpose¶

This module provides periodic output of rsyslog internal counters.

The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). Be sure that your configuration records the counter messages (default is syslog.=info). Besides logging to the regular syslog stream, the module can also be configured to write statistics data into a (local) file.

When logging to the regular syslog stream, impstats records are emitted just like regular log messages. As such, counters increase when processing these messages. This must be taken into consideration when testing and troubleshooting.

Note that loading this module has some impact on rsyslog performance. Depending on settings, this impact may be noticeable for high-load environments, but in general the overhead is pretty light.

Note that there is a rsyslog statistics online analyzer available. It can be given a impstats-generated file and will return problems it detects. Note that the analyzer cannot replace a human in getting things right, but it is expected to be a good aid in starting to understand and gain information from the pstats logs.

The rsyslog website has an overview of available rsyslog statistic counters. When browsing this page, please be sure to take note of which rsyslog version is required to provide a specific counter. Counters are continuously being added, and older versions do not support everything.

Notable Features¶

Statistic Counter
Push Mode (VictoriaMetrics / Prometheus Remote Write)

Choose Your Mode¶

impstats supports three common operating modes:

Local emission only (syslog and/or file)
Push mode only (Remote Write endpoint)
Local emission and push mode together

Use this page for module reference and parameters. For task-oriented workflows and troubleshooting, use:

Push Mode (VictoriaMetrics / Prometheus Remote Write)¶

impstats can push counters directly to VictoriaMetrics (and other Prometheus Remote Write compatible backends) on every impstats interval. This is enabled by setting push.url to a Remote Write endpoint such as http://localhost:8428/api/v1/write.

When push mode is enabled, impstats reads internal counters directly from stats objects and sends them as Prometheus Remote Write payloads (application/x-protobuf with snappy compression) over HTTP.

Push mode is independent of local emission format and destination. You can still use syslog/file output and any supported format for local records while push mode runs in parallel.

Key notes:

Push mode is available only in builds configured with --enable-impstats-push.
Project-provided rsyslog packages are built with impstats push support. For package library locations, see rsyslog package downloads on rsyslog.com.
TLS options (push.tls.*) apply only when push.url uses https://.
push.tls.certfile and push.tls.keyfile must be configured together.
Optional batch controls (push.batch.maxBytes and push.batch.maxSeries) split large Remote Write payloads.
For naming, labels, retries, and limitations details, see FAQ: impstats Push Mode (Remote Write) and Tutorial: impstats Push to VictoriaMetrics.

Configuration Parameters¶

The configuration parameters are grouped below by use case.

Note

Parameter names are case-insensitive; camelCase is recommended for improved readability.

Note

This module supports module parameters, only.

Core Module Parameters (Local Emission)¶

Parameter	Summary
interval	Configures how often, in seconds, impstats emits statistics messages.
facility	Selects the numeric syslog facility value used for generated statistics messages.
severity	Sets the syslog severity value that impstats assigns to emitted messages.
resetcounters	Controls whether impstats resets counters after emitting them so they show deltas.
format	Specifies which output format impstats uses for emitted statistics records.
log.syslog	Enables or disables sending statistics to the regular syslog stream.
log.file	Writes impstats statistics records to the specified local file in addition to other outputs.
log.file.overwrite	If set to “on”, the statistics log file specified by log.file is overwritten with each emission instead of being appended to.
ruleset	Binds impstats-generated messages to the specified ruleset for further processing.
bracketing	Adds BEGIN and END marker messages so post-processing can group statistics blocks.

Push Mode Parameters (Remote Write)¶

Parameter	Summary
push.url	Specifies the Prometheus Remote Write endpoint URL to enable impstats push.
push.labels	Adds static labels (`key=value`) to every pushed metric.
push.timeout.ms	Sets the HTTP request timeout for Remote Write push (milliseconds).
push.label.instance	Controls whether impstats adds `instance=<hostname>` to pushed metrics.
push.label.job	Adds a `job=<value>` label to pushed metrics.
push.label.origin	Controls whether impstats adds the stats origin as an `origin` label.
push.label.name	Controls whether impstats adds the stats object name as a `name` label.
push.batch.maxBytes	Sets an approximate maximum protobuf size per Remote Write request.
push.batch.maxSeries	Limits the number of time series per Remote Write request.
push.tls.cafile	Specifies the CA bundle used to verify the Remote Write TLS certificate.
push.tls.certfile	Specifies the client certificate file used for mutual TLS.
push.tls.keyfile	Specifies the client private key file used for mutual TLS.
push.tls.insecureSkipVerify	Disables TLS certificate verification for Remote Write (testing only).

Statistic Counter¶

The impstats plugin gathers some internal statistics. They have different names depending on the actual statistics. Obviously, they do not relate to the plugin itself but rather to a broader object – most notably the rsyslog process itself. The “resource-usage” counter maintains process statistics. They base on the getrusage() system call. The counters are named like getrusage returned data members. So for details, looking them up in “man getrusage” is highly recommended, especially as value may be different depending on the platform. A getrusage() call is done immediately before the counter is emitted. The following individual counters are maintained:

utime - this is the user time in microseconds (thus the timeval structure combined)
stime - again, time given in microseconds
maxrss
minflt
majflt
inblock
outblock
nvcsw
nivcsw
openfiles - number of file handles used by rsyslog; includes actual files, sockets and others

Format¶

type	default	mandatory	`obsolete legacy` directive
word	legacy	no	none

Added in version 8.16.0.

Specifies the format of emitted stats messages. The default of “legacy” is compatible with pre v6-rsyslog. The other options provide support for structured formats (note the “cee” is actually “project lumberjack” logging).

The json-elasticsearch format supports the broken ElasticSearch JSON implementation. ES 2.0 no longer supports valid JSON and disallows dots inside names. The “json-elasticsearch” format option replaces those dots by the bang (“!”) character. So “discarded.full” becomes “discarded!full”.

The zabbix format supports arrayed-JSON objects with a single JSON object per pstats emission to disk or syslog. This format should be compatible with most JSON parsers from other monitoring products. log.file is highly recommended as log.syslog may encounter message truncation problems if the emission is large. If you must use log.syslog, it’s recommended to monitor pstats for truncation and increase $MaxMessageSize at the top of your main rsyslog configuration file.

Options: json/json-elasticsearch/cee/legacy/prometheus/zabbix

Caveats/Known Bugs¶

This module MUST be loaded right at the top of rsyslog.conf, otherwise stats may not get turned on in all places.
When using the format “zabbix”, it is not recommended to use logSyslog=”on”. This can cause message truncation in stats.

Examples¶

Load module, send stats data to syslog stream¶

This activates the module and records messages to /var/log/rsyslog-stats in 10 minute intervals:

module(load="impstats"
       interval="600"
       severity="7")

# to actually gather the data:
syslog=debug/var/log/rsyslog-stats

Load module, format the output to Zabbix, and output to a local file¶

module(load="impstats"
       interval="60"
       severity="7"
       format="zabbix"
       logSyslog="off"
       logFile="/var/log/stats.log")

Load module, send stats data to local file¶

Here, the default interval of 5 minutes is used. However, this time, stats data is NOT emitted to the syslog stream but to a local file instead.

module(load="impstats"
       interval="600"
       severity="7"
       logSyslog="off"
       # need to turn log stream logging off!
       logFile="/path/to/local/stats.log")

Load module, send stats data to local file and syslog stream¶

Here we log to both the regular syslog log stream as well as a file. Within the log stream, we forward the data records to another server:

module(load="impstats"
       interval="600"
       severity="7"
       logFile="/path/to/local/stats.log")

syslog=debug@central.example.net

Load module with Remote Write push mode (VictoriaMetrics)¶

This keeps local output and additionally pushes the same interval counters to VictoriaMetrics using Prometheus Remote Write:

module(load="impstats"
       interval="30"
       format="json"
       logFile="/var/log/rsyslog-stats.log"
       push.url="http://localhost:8428/api/v1/write"
       push.labels=["env=prod", "cluster=edge"]
       push.label.origin="on"
       push.label.name="on")

Explanation of output¶

Example output for illustration:

Sep 17 11:43:49 localhost rsyslogd-pstats: imuxsock: submitted=16
Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2

Explanation:

All objects are shown in the results with a separate counter, one object per line.

Line 1: shows details for

imuxsock, an object
submitted=16, a counter showing that 16 messages were received by the imuxsock object.

Line 2: shows details for the main queue:

main Q, an object
size, messages in the queue
enqueued, all received messages thus far
full, how often was the queue was full
maxqsize, the maximum amount of messages that have passed through the queue since rsyslog was started

impstats: Generate Periodic Statistics of Internal Counters¶

Purpose¶

Notable Features¶

Choose Your Mode¶

Push Mode (VictoriaMetrics / Prometheus Remote Write)¶

Configuration Parameters¶

Core Module Parameters (Local Emission)¶

Push Mode Parameters (Remote Write)¶

Statistic Counter¶

Format¶

Caveats/Known Bugs¶

Examples¶

Load module, send stats data to syslog stream¶

Load module, format the output to Zabbix, and output to a local file¶

Load module, send stats data to local file¶

Load module, send stats data to local file and syslog stream¶

Load module with Remote Write push mode (VictoriaMetrics)¶

Explanation of output¶

See Also¶