Log levels and diagnostics

Log levels

[general] logLevel sets how much Collectord logs: trace, debug, info (the default), warn, error or fatal. To trace one part of Collectord without the noise of the rest, raise only its level with logLevel.<subsystem>. For example, to see why a Docker container’s logs are not picked up:

1[general]
2logLevel = info
3logLevel.collectord.pipeline.input.docker = trace

Subsystem names are dotted paths, and they nest: logLevel.collectord.pipeline.output = warn quiets every output, and a longer name below it wins.

Collectord applies a changed log level without a restart. It checks its configuration files every second (every ten seconds on macOS) and, when one changes, reloads the log levels and the license; a SIGHUP reloads them at once. Every other setting still takes a restart.

Before 26.10, SIGHUP stopped Collectord. A script that sends it to stop Collectord should send SIGTERM or SIGINT.

On Kubernetes and OpenShift, edit the ConfigMap in place to raise a level on running pods: they pick up the change once Kubernetes refreshes the mounted files, usually within a minute or two, until the next helm upgrade puts the chart’s ConfigMap back. A helm upgrade with changed values restarts the pods instead, since the chart rolls them on a configuration change. The chart’s configuration.general has no key for logLevel.<subsystem>, flightRecorder or shutdownTimeout: add them in a file of their own, which Collectord reads after the shipped ones:

1collectord:
2  configMap:
3    allAdditionalFiles:
4      "101-general.conf": |
5        [general]
6        logLevel.collectord.pipeline.output.splunk = debug

The DaemonSet and the add-on carry an annotation that drops Collectord’s own TRACE and DEBUG lines before they reach Splunk, so read a raised level with kubectl logs (oc logs on OpenShift).

Verify and collect a diagnostic bundle

collectord verify checks every input and output in the configuration - the license, the connection to Splunk or Elasticsearch, the paths it reads - and prints OK or the error for each. collectord diag builds the bundle support asks for: performance and memory profiles, Collectord’s metrics, the host’s details and the configuration, with the output’s URL and token removed. Each product’s troubleshooting page has the exact commands: Kubernetes, OpenShift, Docker, Linux, macOS, Elasticsearch and OpenSearch and syslog.

When Collectord crashes, the trace often never reaches Splunk, because the crash races Collectord’s own log shipping. Collectord also writes it to crash.txt under its data path, keeps it as crash-previous.txt when it starts again, and every diag bundle includes both, so a diag taken after the restart still shows the crash.

Flight recorder

A stall that already happened is gone before a diag can record it. With [general] flightRecorder = true, Collectord keeps the last ten seconds or so of its execution trace in memory (up to 8 MiB) and adds it to every diag bundle as flight.trace. It is off by default, since tracing all the time costs a few percent of CPU: turn it on when support asks, and send the next diag after the problem shows again. Collectord reads it when it starts, so it takes a restart.

Shutdown timeout

When Collectord stops, it waits until every input and output has finished, so an output still sending to an unreachable Splunk can hold the shutdown. [general] shutdownTimeout bounds that wait, for example shutdownTimeout = 30s: past it, Collectord logs which part did not finish and exits. Unset, it waits for everything. A second SIGTERM or SIGINT during the shutdown stops the wait at once.