Reduce log volume

When a workload writes the same line over and over, every copy costs license and disk. Collectord has three features for it, all off until you switch them on:

Start with the report, on the workloads the Splunk Usage Report’s Breakdown tab shows cost the most: it shows which of their lines are worth grouping or folding.

What each costs

The savings were measured on 18 days of one OpenShift cluster’s container logs (6.0 million lines, 1.24 GB) at the defaults, and depend on how your workloads log. Collectord’s cost was measured on Collectord for Docker reading 10 containers that each write 1,000 lines of 1 KB a second, no two of them repeating or sharing a pattern: with all three features on, 0.43 of a core instead of 0.36, and 38 MiB instead of 22.5 MiB. Each feature keeps at most 256 KB of state per stream, so memory grows with the number of streams, not lines: with 100 containers writing the same 10,000 lines a second, all three took Collectord to 198 MiB instead of 45 MiB, about 1.5 MiB per stream, while its CPU grew by the same 0.07 of a core.

Where you switch them on

Each feature has one switch, for one workload or for all of them:

• One workload: an annotation on a pod, a workload, a namespace or a Configuration on Kubernetes and OpenShift, or a container label on Docker and Podman - collectord.io/logs-repeat-enabled=true, collectord.io/logs-fold-enabled=true or collectord.io/logs-patterns-enabled=true. Put <container>-- in front of the key for one container of a pod, or stdout- or stderr- for one stream. For the log files on a volume, use collectord.io/volume.<n>-logs-repeat-enabled and its fold and patterns twins. Set to false, the annotation turns the feature off for a workload when the configuration turns it on for all.
• Every workload, or a host’s logs: repeatEnabled = true, foldEnabled = true or patternsEnabled = true in the configuration - [input.files] for container logs, [input.app_logs] for volume logs, an [input.files::<name>] section for a host’s log files, and [input.journald] for the journal. In the Helm charts, the same keys sit under collectord.configuration.inputs in containerLogs, applicationLogs, hostLogs.<name> and journald.

Group repeated lines

When a stream writes the same line again and again - a health check, a “waiting for leader” loop, a reconnect message - Collectord sends the first line at once and holds the repeats after it for up to a minute. Then it sends one event in their place: the first repeat as it was written, with collectord_repeat_count, the lines the event stands for, and collectord_repeat_span, the seconds from its earliest line to its latest. Only the copies are left out.

Lines are compared byte for byte, leaving out only the line’s own timestamp (at its start, or under a time key such as "ts": or time=). A line is a repeat of any of the stream’s last 256 distinct lines, not only of the line before it, so the cycles of a failing reconcile loop group too.

To group the repeated lines of every workload in a namespace:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: payments
5  annotations:
6    collectord.io/logs-repeat-enabled: "true"

Measured on 18 days of a cluster’s container logs, grouping at the defaults left out 17.7% of the bytes, and 24.5% with a five-minute hold. It does not reach a crash-looping pod, whose every restart is a new container, or lines that carry a request ID.

A grouped event carries its first repeat’s time and arrives up to the hold later. A restart of Collectord can send up to one hold of a grouped stream’s lines again; none are lost. Leave grouping off for streams whose identical rows are the data, such as nvidia-smi dmon samples. Into a syslog output, a stream groups only when its syslog.format has a {{collectord_repeat_count}} placeholder; Collectord says so in its log otherwise.

Fold storms of alike lines

When a stream writes the same message many times a second with different numbers in it - a retry loop, a chatty access log, every request to a dependency that is away - the lines are alike rather than equal, and grouping does not catch them. Folding does, at a price: the folded lines are not sent.

Two lines are alike when they differ only in their numbers - ids, addresses, ports, counts, durations, times - and come from the same stream and the same writer. Each minute, the first 60 lines of one pattern go out as they are. Collectord drops the rest, and when the minute is over it sends one summary in their place: the first line it dropped, with collectord_fold_count, the lines it stands for, and collectord_fold_bytes, their size. Lines that match (?i)error|exception|fatal|panic are never folded, and neither are lines you exempted from the thruput caps. Measured on 18 days of a cluster’s container logs, the defaults folded 15% of the bytes.

To fold one Docker container’s lines past 20 a minute per pattern:

1docker run -d \
2    --label 'collectord.io/logs-fold-enabled=true' \
3    --label 'collectord.io/logs-fold-maxperminute=20' \
4    payments-api:2.4

A summary carries its first dropped line’s time and goes out once its minute is over. A restart or a crash loses the count of a minute not yet over, never a line Collectord was meant to send. Into a syslog output, a stream folds only when its syslog.format names {{collectord_fold_count}}.

Report the costliest log patterns

The pattern report tells you which lines cost a workload the most, without changing its logs. Every 15 minutes, each stream sends its 20 heaviest patterns - its lines alike but for their numbers - as events of their own. Each report event carries the pattern and an example line as JSON in its message, and collectord_pattern_lines, collectord_pattern_bytes and collectord_pattern_seconds (the interval it covers) as fields. It goes to the same index and output as the lines it counts, under the input’s source type with _patterns after it - kubernetes_logs_patterns, kubernetes_host_logs_patterns, docker_logs_patterns, linux_host_logs_patterns - so no search that counts log lines finds it among them.

A report costs license of its own, and as much for a quiet stream as for a busy one: about 300 bytes per pattern it carries. Measured on a demo cluster, a stream averaged 1.6 KB of reports every 15 minutes, about 150 KB a day, which is more than a quiet workload’s own logs. Switch it on for the workloads that cost the most, not for everything. To report on one namespace’s workloads:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: payments
5  annotations:
6    collectord.io/logs-patterns-enabled: "true"

In a log file that several processes write, such as syslog, extractionWriterFields in its [input.files::<name>] section names the extracted fields that tell the writers apart, so each writer’s lines make patterns of their own. The shipped syslog sections set it to syslog_hostname, syslog_component, syslog_pid. On macOS, only the [input.files::<name>] log files report; the unified log has no pattern report. A stream sending into a syslog output does not report.

Monitoring Kubernetes, OpenShift, Docker, Linux and macOS show the reports in Setup › Splunk Usage Report, under the Log patterns view: the heaviest patterns of the period with who wrote them, the lines and megabytes they sent, their share of what the reports counted, the hosts, a trend and an example line. The view stays empty until a stream reports. For your own searches, each app’s macro_<app>_log_patterns macro finds the reports:

1`macro_kubernetes_log_patterns`
2| stats sum(collectord_pattern_bytes) as bytes, sum(collectord_pattern_lines) as lines by pattern
3| sort - bytes

If you set patternsType, add your source type to the macro.

Count lines, not events

Once a stream groups or folds, one event can stand for many lines, so count lines rather than events:

1`macro_kubernetes_logs`
2| stats sum(eval(coalesce(collectord_repeat_count, collectord_fold_count, 1))) as lines by kubernetes_namespace

The monitoring apps’ dashboards and Alert Catalog templates already count this way, and they alias collectord_repeat_count and collectord_fold_count to their dotted spellings, collectord.repeat.count and collectord.fold.count. In Elasticsearch and OpenSearch, a sum aggregation of collectord.repeat.count with "missing": 1 counts the lines of a grouped stream; the shipped index templates map the repeat and fold fields as numbers.

Grouped events arrive up to the hold after their first line, and fold summaries once their minute is over, so give a scheduled search over such a stream that much lag.