Annotations

Annotations tell Collectord to treat one namespace, workload or pod differently, without touching the global configuration.

Use them to:

• route data to a different index or Splunk output;
• forward application logs that don’t go to stdout;
• mask, hash or drop sensitive values;
• extract fields and fix multi-line stack traces;
• scrape Prometheus endpoints.

This page shows the annotations worth an example. The full list of every annotation lives in the Annotations reference.

Running more than one Collectord on the same cluster? Scope annotations to one Collectord instance with [general]annotationsSubdomain. Annotations under collectord.collectord.io/{annotation} apply to every Collectord instance, whatever its subdomain.

Overriding indexes

Use these annotations to send a namespace, workload or pod’s data to a different Splunk index than the cluster default: for example, to give a team its own index for chargeback.

• collectord.io/index is the catch-all. It sets the events index of everything the workload sends: logs, the stats and Prometheus metrics Collectord sends to an events index, and, set on a namespace, the namespace’s events and objects.
• A datatype’s own annotation gives finer control:
  • collectord.io/logs-index for container logs;
  • collectord.io/stats-index for container metrics;
  • collectord.io/procstats-index for process stats;
  • collectord.io/events-index for events, which can only be routed at the namespace level.

Stats and Prometheus metrics sent to a Splunk metrics index take their index another way: see Events index and metrics index. To keep a team’s data in indexes only that team writes to, see Giving a team its own indexes.

To send everything from a team’s namespace to one index:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: payments
5  annotations:
6    collectord.io/index: openshift_payments

Everything from this namespace that goes to an events index now lands in openshift_payments: container logs, application logs, events, the namespace’s objects, pod, container, process and network stats, and the pods’ Prometheus metrics.

A change to a pod’s own annotations applies as soon as Collectord sees the pod’s update. A change to the annotations of a namespace, a Deployment or a Collectord Configuration reaches the pods it applies to within about half a minute. Collectord watches those kinds (the watch.<name> values under [general.kubernetes]) and re-reads every pod when one changes.

A change to any other owner, such as a DeploymentConfig, StatefulSet, DaemonSet or CronJob, applies when Collectord next re-lists the pods, within [general.kubernetes]/timeout (30 minutes by default). To get the fast path for DeploymentConfigs, add watch.deploymentconfig = apps.openshift.io/v1/deploymentconfig under [general.kubernetes]. To apply a change immediately, recreate the pod (waiting [general.kubernetes]/metadataTTL after the change) or restart Collectord.

For indexing performance, a common pattern keeps logs and stats in separate events indexes. Stats indexes are typically much smaller and benefit from a different retention policy:

 1apiVersion: v1
 2kind: Namespace
 3metadata:
 4  name: payments
 5  annotations:
 6    collectord.io/index: openshift_payments
 7    collectord.io/procstats-index: openshift_payments_stats
 8    collectord.io/netstats-index: openshift_payments_stats
 9    collectord.io/nettable-index: openshift_payments_stats
10    collectord.io/stats-index: openshift_payments_stats

Logs and events still go to openshift_payments; pod, container, process, and network stats go to openshift_payments_stats.

source, type, and host follow the same pattern: collectord.io/source, collectord.io/logs-source, and so on.

collectord.io/logs-index only overrides container logs. To override application logs, use collectord.io/index (everything) or collectord.io/volume.{N}-logs-index (per-volume).

Events index and metrics index

Logs, events and objects always go to an events index. Stats and Prometheus metrics go to an events index, a metrics index, or both, as the input’s indexType says, or [general] defaultIndexType when the input sets none. Their metrics index is chosen apart from the events index.

Each annotation in the table is collectord.io/<name>. An index or output a row leaves unset falls back to collectord.io/index and collectord.io/output.

A stats record bound for a metrics index takes, in order:

1. On an output of the workload’s own, that output’s defaultMetricsIndex (since 26.10). An output of its own is an [output.splunk::<name>] or a SplunkOutput that collectord.io/output (or stats-output, …) names. The input’s metricsIndex and metricsOutput in the configuration are for the cluster’s shared output and don’t apply.
2. Otherwise the input’s metricsIndex in the configuration, else the output’s defaultMetricsIndex.
3. With neither: under indexType = metrics, the row’s events-index annotation, as before 26.10; under indexType = both, the HEC token’s default index.

A Prometheus endpoint’s prometheus.{N}-metricsIndex and prometheus.{N}-metricsOutput come before all of these.

Keep a cluster’s stats on one kind of index; indexType = both is for the move from one to the other. A team then needs one events index, plus one metrics index when the stats go there.

Overriding index, source and type for specific events

One container can write several kinds of log lines to the same stream: an nginx container writes both access logs and error logs, for example. Override pipes split them at ingest time. Each override pipe matches a regex and rewrites source, type, or index only for matching events.

For an nginx container writing:

1172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
22018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
3172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

To send only the access-log lines (those starting with an IPv4 address) to a custom source:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-override.1-match: ^(\d{1,3}\.){3}\d{1,3}
 7    collectord.io/logs-override.1-source: /openshift/nginx/web-log
 8spec:
 9  containers:
10  - name: nginx
11    image: nginxinc/nginx-unprivileged

The error-log line keeps the default container-log source; everything matching the IP regex gets the new one:

1source                    | event
2------------------------------------------------------------------------------------------------------------------------
3/openshift/nginx/web-log  | 172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
4/openshift/550...stderr   | 2018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
5/openshift/nginx/web-log  | 172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

Replace patterns in events

Replace pipes rewrite parts of a log line before it reaches Splunk. Use them to mask sensitive data (PII, tokens, IPs) or to strip noise. Each pipe is a pair of annotations grouped by number:

• collectord.io/logs-replace.{N}-search is the regex.
• collectord.io/logs-replace.{N}-val is the replacement. Use $1 or ${name} in it to reference capture groups.

Pipes apply in alphabetical order of their numbers, so you can chain them: replace.1 runs before replace.2, and replace.10 before replace.2.

Collectord uses Go’s regexp library - see Package regexp and re2 syntax. regex101.com is great for testing (set the Flavor to golang).

The examples below use nginx access logs:

1172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 1. Replacing IPv4 addresses with X.X.X.X

To fully mask client IPs:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-replace.1-search: (\d{1,3}\.){3}\d{1,3}
 7    collectord.io/logs-replace.1-val: X.X.X.X
 8spec:
 9  containers:
10  - name: nginx
11    image: nginxinc/nginx-unprivileged

Splunk receives:

1X.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

To keep the first octet, for partial geolocation while still anonymizing, capture it with a named group:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-replace.1-search: (?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}
 7    collectord.io/logs-replace.1-val: ${IPv4p1}.X.X.X
 8spec:
 9  containers:
10  - name: nginx
11    image: nginxinc/nginx-unprivileged

Result:

1172.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2172.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3172.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 2. Dropping messages

Replacing a whole line with the empty string drops the event entirely. Below, the first pipe drops noisy successful GET requests, and the second masks IPs on whatever’s left:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
 7    collectord.io/logs-replace.1-val: ''
 8    collectord.io/logs-replace.2-search: '(\d{1,3}\.){3}\d{1,3}'
 9    collectord.io/logs-replace.2-val: 'X.X.X.X'
10spec:
11  containers:
12  - name: nginx
13    image: nginxinc/nginx-unprivileged

replace.1 drops the success line first, then replace.2 masks IPs on the remaining errors:

1X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
2X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 3. Whitelisting the messages

When the logs you care about are a small share of the volume, a whitelist is easier than a blacklist. With collectord.io/logs-whitelist, Collectord forwards only the lines matching the regex and drops everything else:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-whitelist: '((DELETE)|(POST))$'
 7spec:
 8  containers:
 9  - name: nginx
10    image: nginxinc/nginx-unprivileged

Hashing values in logs

When you need to correlate events by a sensitive field but can’t store the raw value, hash it instead of replacing it. A value always hashes the same way: searching for the hash of a known IP finds every line containing that IP, and the IP itself never reaches Splunk.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
 7    collectord.io/logs-hashing.1-function: 'fnv-1a-64'
 8spec:
 9  containers:
10  - name: nginx
11    image: nginxinc/nginx-unprivileged

A line that originally read:

1172.17.0.1 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

becomes, with fnv-1a-64:

1gqsxydjtZL4 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

Collectord supports both fast non-cryptographic hashes (FNV, CRC, Adler) and cryptographic ones (MD5, the SHA family). Pick the cheapest one that meets your security requirements: non-cryptographic hashes are fine for correlation, but don’t rely on them for security. The benchmarks below are nanoseconds per operation, hashing the two IP addresses in source: 127.0.0.1, destination: 10.10.1.99:

 1| Function          | ns / op |
 2-------------------------------
 3| adler-32          |    1713 |
 4| crc-32-ieee       |    1807 |
 5| crc-32-castagnoli |    1758 |
 6| crc-32-koopman    |    1753 |
 7| crc-64-iso        |    1739 |
 8| crc-64-ecma       |    1740 |
 9| fnv-1-64          |    1711 |
10| fnv-1a-64         |    1711 |
11| fnv-1-32          |    1744 |
12| fnv-1a-32         |    1738 |
13| fnv-1-128         |    1852 |
14| fnv-1a-128        |    1836 |
15| md5               |    2032 |
16| sha1              |    2037 |
17| sha256            |    2220 |
18| sha384            |    2432 |
19| sha512            |    2516 |

Escaping terminal sequences, including terminal colors

Containers attached to a TTY often emit ANSI color codes, which look like garbage in Splunk. One annotation fixes it. This example deliberately runs ls --color=auto with tty: true:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: ubuntu-shell
 5spec:
 6  containers:
 7  - name: ubuntu
 8    image: ubuntu
 9    tty: true
10    command: [/bin/sh, -c,
11             'while true; do ls --color=auto /; sleep 5; done;']

Without intervention, Splunk shows:

1[01;34mboot[0m  [01;34metc[0m  [01;34mlib[0m   [01;34mmedia[0m  [01;34mopt[0m  [01;34mroot[0m  [01;34msbin[0m  [01;34msys[0m  [01;34musr[0m
2[0m[01;34mbin[0m   [01;34mdev[0m  [01;34mhome[0m  [01;34mlib64[0m  [01;34mmnt[0m  [01;34mproc[0m  [01;34mrun[0m   [01;34msrv[0m  [30;42mtmp[0m  [01;34mvar[0m

Add collectord.io/logs-escapeterminalsequences: 'true' and Collectord strips them before forwarding:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: ubuntu-shell
 5  annotations:
 6    collectord.io/logs-escapeterminalsequences: 'true'
 7spec:
 8  containers:
 9  - name: ubuntu
10    image: ubuntu
11    tty: true
12    command: [/bin/sh, -c,
13             'while true; do ls --color=auto /; sleep 5; done;']

Now Splunk shows clean output:

1bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
2boot  etc  lib   media  opt  root  sbin  sys  usr

If most of your containers emit color codes, change the global default in the Collectord configuration:

• [input.files]/stripTerminalEscapeSequences sets whether Collectord strips them by default (false by default).
• [input.files]/stripTerminalEscapeSequencesRegex sets which sequences match.

Extracting fields from the container logs

Field extraction at ingest time pulls structured values out of unstructured log lines: the timestamp, an IP address, a request path. Splunk indexes them, so it can search them as fields instead of scanning _raw. That makes searches dramatically faster on high-volume indexes.

The examples use the same nginx access logs:

1172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

By default, the last unnamed capture group becomes the event message (_raw), so use a single unnamed group in the pattern to avoid surprises. To pick a different group as the message, set collectord.io/logs-extractionMessageField.

Example 1. Extracting the timestamp

When the container’s own timestamp is more accurate than ingest time (clock skew, batched logs, replay), extract it and use it as _time. Specify the regex, the named group containing the timestamp, and the format.

Collectord uses Go’s time parser, which uses the reference date Mon Jan 2 15:04:05 MST 2006 to describe formats - see Go documentation.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-extraction: '^(.*\[(?P<timestamp>[^\]]+)\].+)$'
 7    collectord.io/logs-timestampfield: timestamp
 8    collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
 9spec:
10  containers:
11  - name: nginx
12    image: nginxinc/nginx-unprivileged

The event’s _time in Splunk now matches the timestamp inside the log line.

Available since Collectord version 5.24.440 For unix epoch timestamps, use the format @unixtimestamp.

Example 2. Extracting the fields

Once you’ve moved the timestamp to _time, you usually don’t want it duplicated in _raw. Extract additional fields and let the rest fall into the message:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/logs-extraction: '^(?P<ip_address>[^\s]+) .* \[(?P<timestamp>[^\]]+)\] (.+)$'
 7    collectord.io/logs-timestampfield: timestamp
 8    collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
 9spec:
10  containers:
11  - name: nginx
12    image: nginxinc/nginx-unprivileged

Splunk now has ip_address as an indexed field, the parsed _time, and a tighter _raw:

1ip_address | _time               | _raw
2-----------|---------------------|-------------------------------------------------
3172.17.0.1 | 2018-08-31 21:11:26 | "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
4172.17.0.1 | 2018-08-31 21:11:32 | "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
5172.17.0.1 | 2018-08-31 21:11:35 | "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Defining Event pattern

collectord.io/logs-eventpattern sets where one log event ends and the next begins. The default in collectord configuration is ^[^\s]: any line that doesn’t start with whitespace begins a new event. That handles most stack traces, whose continuation lines are indented. It breaks for log formats whose continuation lines start in column 0.

A common case is Java and Elasticsearch errors, whose call stack header doesn’t begin with whitespace. The pod below deliberately misconfigures Elasticsearch (s-node instead of single-node) to get a multi-line stack trace:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: elasticsearch-pod
 5spec:
 6  containers:
 7  - name: elasticsearch
 8    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
 9    env:
10    - name: discovery.type
11      value: s-node

The output looks like:

 1[2018-08-31T22:44:56,433][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/92] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
 2[2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
 3org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
 4	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]
 5	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
 6	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
 7	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
 8	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
 9	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]
10	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
11Caused by: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
12	at org.elasticsearch.discovery.DiscoveryModule.<init>(DiscoveryModule.java:129) ~[elasticsearch-6.4.0.jar:6.4.0]
13	at org.elasticsearch.node.Node.<init>(Node.java:477) ~[elasticsearch-6.4.0.jar:6.4.0]
14	at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
15	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
16	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
17	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
18	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
19	... 6 more
20[2018-08-31T22:44:56,892][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

With the default pattern, the warning line [2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] and its entire stack trace get split into separate events.

Tell Collectord that every event in this container starts with [:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: elasticsearch-pod
 5  annotations:
 6    collectord.io/logs-eventpattern: '^\['
 7spec:
 8  containers:
 9  - name: elasticsearch
10    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
11    env:
12    - name: discovery.type
13      value: s-node

Three limits bound a joined event. By default Collectord:

• joins the lines of an event written within 100ms of its first line (maxInterval);
• sends the event once it is 1s old and the log has nothing more to read (maxWait);
• caps a combined event at 1MB (maxSize).

Both timers count from the event’s first line, not from the previous line, so if a workload’s events still split, raise both. [pipe.join] in the Collectord configuration sets them for every pod.

To set them for one pod, annotate it with collectord.io/logs-eventPatternMaxInterval, collectord.io/logs-eventPatternMaxWait and collectord.io/logs-eventPatternMaxSize; each one you leave out keeps the [pipe.join] value.

Application Logs

Some applications can’t send everything to stdout and stderr; they write files inside the container. Audit logs, slow-query logs, GC logs, and anything that needs to survive a process restart typically end up on disk. Collectord reads them without a sidecar: mount a volume for them and add an annotation that names it.

In the example below, a postgres container writes its detailed logs to /var/log/postgresql. An emptyDir volume named logs is mounted there, and the annotation collectord.io/volume.1-logs-name: 'logs' tells Collectord to scan that volume for log files. By default Collectord picks up the files matching the global glob *.log*; override it per volume with collectord.io/volume.{N}-logs-glob.

For a container with several log directories, group the settings by number: collectord.io/volume.1-logs-name, collectord.io/volume.2-logs-name, and so on. Two groups can also name the same volume to give different files on it their own settings: see Example 3.

For a longer walk-through, see the Complete guide for forwarding application logs from Kubernetes and OpenShift environments to Splunk.

Example 1. Forwarding application logs

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: postgres-pod
 5  annotations:
 6    collectord.io/volume.1-logs-name: 'logs'
 7spec:
 8  containers:
 9  - name: postgres
10    image: postgres
11    command:
12      - docker-entrypoint.sh
13    args:
14      - postgres
15      - -c
16      - logging_collector=on
17      - -c
18      - log_min_duration_statement=0
19      - -c
20      - log_directory=/var/log/postgresql
21      - -c
22      - log_min_messages=INFO
23      - -c
24      - log_rotation_age=1d
25      - -c
26      - log_rotation_size=10MB
27    volumeMounts:
28      - name: data
29        mountPath: /var/lib/postgresql/data
30      - name: logs
31        mountPath: /var/log/postgresql/
32  volumes:
33  - name: data
34    emptyDir: {}
35  - name: logs
36    emptyDir: {}

Each event’s source includes the volume name and file - for example, psql_logs:postgresql-2018-08-31_232946.log:

 12018-08-31 23:31:02.034 UTC [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
 2	  c.relname as "Name",
 3	  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
 4	  pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
 5	FROM pg_catalog.pg_class c
 6	     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
 7	WHERE c.relkind IN ('r','p','')
 8	      AND n.nspname <> 'pg_catalog'
 9	      AND n.nspname <> 'information_schema'
10	      AND n.nspname !~ '^pg_toast'
11	  AND pg_catalog.pg_table_is_visible(c.oid)
12	ORDER BY 1,2;
132018-08-31 23:30:53.490 UTC [124] FATAL:  role "postgresql" does not exist

Example 2. Forwarding application logs with fields extraction and time parsing

Every annotation that works for container logs has a volume.{N}- equivalent for application logs: field extraction, replace patterns, index, source and host overrides, sampling, throttling. Below, the postgres timestamp is extracted and removed from _raw:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: postgres-pod
 5  annotations:
 6    collectord.io/volume.1-logs-name: 'logs'
 7    collectord.io/volume.1-logs-extraction: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} [^\s]+) (.+)$'
 8    collectord.io/volume.1-logs-timestampfield: 'timestamp'
 9    collectord.io/volume.1-logs-timestampformat: '2006-01-02 15:04:05.000 MST'
10spec:
11  containers:
12  - name: postgres
13    image: postgres
14    command:
15      - docker-entrypoint.sh
16    args:
17      - postgres
18      - -c
19      - logging_collector=on
20      - -c
21      - log_min_duration_statement=0
22      - -c
23      - log_directory=/var/log/postgresql
24      - -c
25      - log_min_messages=INFO
26      - -c
27      - log_rotation_age=1d
28      - -c
29      - log_rotation_size=10MB
30    volumeMounts:
31      - name: data
32        mountPath: /var/lib/postgresql/data
33      - name: logs
34        mountPath: /var/log/postgresql/
35  volumes:
36  - name: data
37    emptyDir: {}
38  - name: logs
39    emptyDir: {}

The timestamp moves to _time, and _raw no longer carries the redundant prefix:

 1_time               | _raw
 22018-08-31 23:31:02 | [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
 3                    | 	  c.relname as "Name",
 4                    | 	  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
 5                    | 	  pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
 6                    | 	FROM pg_catalog.pg_class c
 7                    | 	     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
 8                    | 	WHERE c.relkind IN ('r','p','')
 9                    | 	      AND n.nspname <> 'pg_catalog'
10                    | 	      AND n.nspname <> 'information_schema'
11                    | 	      AND n.nspname !~ '^pg_toast'
12                    | 	  AND pg_catalog.pg_table_is_visible(c.oid)
13                    | 	ORDER BY 1,2;
142018-08-31 23:30:53 |  UTC [124] FATAL:  role "postgresql" does not exist

Example 3. A different sourcetype for each file on a volume

One volume can hold several kinds of logs: nginx writes access.log next to error.log, and a JVM writes its application log next to a GC log. You usually want each kind under its own sourcetype, so Splunk applies the right extractions to it. Give each kind its own group, and in every group:

• name the same volume with collectord.io/volume.{N}-logs-name;
• select the group’s files with collectord.io/volume.{N}-logs-glob, or with a regular expression in collectord.io/volume.{N}-logs-match;
• set the sourcetype with collectord.io/volume.{N}-logs-type.

Collectord reads each group on its own, so every other volume.{N}- annotation can differ between the groups as well: index, source, field extraction, event pattern.

Below, nginx writes its logs to an emptyDir volume named logs, and the access and error logs, rotated copies included, each get their own sourcetype:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/volume.1-logs-name: 'logs'
 7    collectord.io/volume.1-logs-glob: 'access.log*'
 8    collectord.io/volume.1-logs-type: 'nginx_access'
 9    collectord.io/volume.2-logs-name: 'logs'
10    collectord.io/volume.2-logs-glob: 'error.log*'
11    collectord.io/volume.2-logs-type: 'nginx_error'
12spec:
13  containers:
14  - name: nginx
15    image: nginxinc/nginx-unprivileged
16    volumeMounts:
17      - name: logs
18        mountPath: /var/log/nginx/
19  volumes:
20  - name: logs
21    emptyDir: {}

1source                          | sourcetype
2/openshift/logs/logs:access.log | nginx_access
3/openshift/logs/logs:error.log  | nginx_error

A file that several groups select is read once, by the lowest-numbered of them (2 before 10), and carries that group’s type, index and source. Groups that send it to different outputs each read it.

So a group for the remaining files needs no selection of its own. Give it the highest number and no glob or match, and the global glob *.log* ([input.app_logs]/glob) gives it every file the other groups leave.

Before 26.10, a file that two groups both selected was read by both, which duplicated its events. So every group on the volume, the one for the remaining files included, needed a selection of its own.

On Volume Database for acknowledgements

Collectord keeps a local database of the files it has already processed, so it doesn’t forward them again after a restart. By default that database lives on the host where Collectord runs.

That is fine for stateless logs, but breaks for PVCs that move between nodes. When a pod with a migrated volume comes back up on a new host, Collectord doesn’t know what it already forwarded and replays the files from the beginning.

Set collectord.io/volume.{N}-logs-onvolumedatabase=true to store the acknowledgement database (.collectord.db) inside the volume itself, so it travels with the data.

This requires write access to /rootfs in the Collectord container. The manifest mounts it writable by default (readOnly: false), so keep it that way.

Volume types

Collectord auto-discovers application logs on three volume types: emptyDir, hostPath (a directory), and persistentVolumeClaim. Two keys in the Collectord configuration say where to look:

• [general.kubernetes]/volumesRootDir for finding emptyDir volumes;
• [input.app_logs]/root for host mounts that may be exposed under a different path inside the Collectord container.

Collectord also reads two newer volume types:

• Generic ephemeral volumes (ephemeral.volumeClaimTemplate), through the PersistentVolumeClaim Kubernetes creates for the pod, named <pod>-<volume>, the same way as a persistentVolumeClaim volume.
• Inline CSI volumes (csi without a claim), from the directory the CSI driver publishes the volume at, including the driver’s mount/ subdirectory, like a CSI-backed claim.

Other volume types aren’t read. An image volume is read-only content the container runtime mounts from an OCI image, so nothing can write logs to it. configMap, secret, projected, downwardAPI and nfs volumes aren’t read either. When a collectord.io/volume.{N}-logs-name annotation names one of them, the Collectord log says which type the volume is, instead of reporting that the pod has no such volume.

Forwarding Prometheus metrics

Many applications already expose Prometheus metrics: nginx, kube-state-metrics, JVM exporters, custom apps with the prometheus client library. Instead of maintaining a static scrape list, let Collectord scrape them per pod. The Collectord addon (deployed as part of collectorforopenshift.yaml) watches for new pods and starts scraping any pod that declares an endpoint in its annotations. Defaults for interval, type, and HTTP timeout come from [input.prometheus_auto].

The minimum is a port. The pod below runs sophos/nginx-prometheus-metrics, an nginx image that exposes its own metrics, and tells Collectord which port and path to scrape:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  labels:
 6    app: webportal
 7  annotations:
 8    collectord.io/prometheus.1-port: '9527'
 9    collectord.io/prometheus.1-path: '/metrics'
10spec:
11  containers:
12  - name: nginx
13    image: sophos/nginx-prometheus-metrics

For details on how Collectord serializes Prometheus metrics, see Prometheus metrics.

The 1- prefix lets a single pod expose several endpoints: prometheus.1-*, prometheus.2-*, and so on. The Annotations reference has the full set of options: the scrape interval, scheme (http/https) with TLS settings, basic auth (username/password), whitelist/blacklist regex filters, and output for routing to a specific HEC.

Forwarding Prometheus metrics in Splunk Metrics Index

Splunk’s metrics index stores time series much more efficiently than the default events index. To send Prometheus metrics there, set indexType: metrics and name the metrics index with metricsIndex. Before you start forwarding, the index must exist with type metrics, and the HEC token of the output the metrics go through must allow it.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  labels:
 6    app: webportal
 7  annotations:
 8    collectord.io/prometheus.1-port: '9527'
 9    collectord.io/prometheus.1-path: '/metrics'
10    collectord.io/prometheus.1-indexType: 'metrics'
11    collectord.io/prometheus.1-metricsIndex: 'openshift_metrics'
12    collectord.io/prometheus.1-metricsOutput: 'splunk::metrics'
13spec:
14  containers:
15  - name: nginx
16    image: sophos/nginx-prometheus-metrics

metricsOutput sends the metrics through another Splunk output: here an [output.splunk::metrics] bound to a HEC token that allows the metrics indexes. The events token then never needs them on its allow-list. Leave metricsOutput out to send the metrics through the pod’s own output. Without metricsIndex, the metrics go to the output’s defaultMetricsIndex (see Metrics index).

metricsIndex and metricsOutput are available since 26.10. Before, the metrics index was named with prometheus.1-index and the output with prometheus.1-output. That still works when the output has no defaultMetricsIndex. When it has one, the output’s defaultMetricsIndex wins over prometheus.1-index; an [output.splunk::<name>] takes [output.splunk]’s unless it sets its own.

Change output destination

By default Collectord forwards everything to Splunk. To drop a container’s data entirely, use collectord.io/output=devnull: the data is still collected, it just isn’t sent anywhere. That covers spammy debug containers and namespaces you don’t care about. To drop only logs and keep metrics and events, use collectord.io/logs-output=devnull.

You can also flip the default. Start Collectord with --env "COLLECTOR__LOGS_OUTPUT=input.files__output=devnull" so logs are dropped by default, then opt pods in with collectord.io/logs-output=splunk. This is the cleanest pattern for clusters with many noisy workloads where only a few teams want logs in Splunk:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  labels:
 6    app: webportal
 7  annotations:
 8    collectord.io/logs-output: 'splunk'
 9spec:
10  containers:
11  - name: nginx
12    image: nginxinc/nginx-unprivileged

When you have several Splunk outputs configured, a prod cluster and a dev cluster for example, pick one with the output suffix (see Support for multiple Splunk clusters):

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  labels:
 6    app: webportal
 7  annotations:
 8    collectord.io/output: 'splunk::prod1'
 9spec:
10  containers:
11  - name: nginx
12    image: nginxinc/nginx-unprivileged

Forwarding logs to multiple Splunk HTTP Event Collector endpoints simultaneously

Sometimes you need the same log line in two places: a security index for SIEM and an apps index for developers, for example. Pass collectord.io/logs-output a comma-separated list of outputs, defined in your ConfigMap as [output.splunk::apps] and [output.splunk::security]. Each event is sent to all listed endpoints. To override the index per output, put it in square brackets: splunk::apps[openshift_logs],splunk::security[openshift_security].

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: audit-logger
 5  labels:
 6    app: audit-logger
 7  annotations:
 8    collectord.io/logs-output: 'splunk::apps[openshift_logs],splunk::security[openshift_security]'
 9spec:
10  containers:
11  - name: nginx
12    image: nginxinc/nginx-unprivileged

Routing OpenShift events to a separate output

collectord.io/output applies to every datatype, and collectord.io/logs-output only to container logs. To route a namespace’s OpenShift events to a different Splunk output, use collectord.io/events-output. It can only be set on namespaces, since Collectord forwards events per namespace.

Use it to keep events in a dedicated index, to power alerting and audit dashboards for example, while pod logs stay in a different index for day-to-day debugging.

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: payments
5  annotations:
6    collectord.io/events-output: 'splunk::audit'

The format matches collectord.io/output. It takes a single output (splunk::audit), a comma-separated list of outputs, and per-output indexes in square brackets (for example splunk::audit[openshift_events_audit],splunk::ops[openshift_events_ops]).

User outputs

In a multi-tenant cluster, the platform team often owns Collectord while app teams own their data routing. A SplunkOutput lets a team define its own Splunk destination without editing the Collectord ConfigMap. Define the SplunkOutput CRD in the team’s namespace:

1apiVersion: "collectord.io/v1"
2kind: SplunkOutput
3metadata:
4  name: splunk-user-output-for-deployment
5spec:
6  token: 1a8b9c3e-7789-4353-821f-15b9662bac99
7  url: https://splunk.example.com:8088/services/collector/event/1.0
8  insecure: true

Then reference it from the workload as splunk::user/<namespace>/<name>:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: long-running
5  annotations:
6    collectord.io/output: splunk::user/default/splunk-user-output-for-deployment
7spec:
8  ...

Available since Collectord version 26.10 The namespace in the name must be the annotated object’s own. A pod, its namespace, or a Configuration that matches the pod can name a SplunkOutput of the pod’s namespace only. Collectord reads the SplunkOutput and its token with its own cluster-wide access, so without this rule a team could send its data with another team’s token into that team’s indexes.

Collectord rejects such an annotation and keeps the output it would have replaced. It logs a warning naming the annotation, and collectord describe lists it under problems. Where one SplunkOutput serves several namespaces on purpose, set crossNamespaceUserOutputs = true under [general.kubernetes] (configuration.general.kubernetes.crossNamespaceUserOutputs: true in the chart values).

Available since Collectord version 26.10 A SplunkOutput also sets its own defaultMetricsIndex, incorrectIndexBehavior and requireExplicitIndex, and takes every setting it does not set from [output.splunk]. Helm installs the SplunkOutput definition only on the first install. On a cluster installed with an earlier chart, apply the current definition first (helm show crds <chart> | oc apply -f -); the flat manifests include it.

Giving a team its own indexes

Splunk, not Collectord, keeps one team’s data out of another team’s indexes. Each team writes through a HEC token of its own, allowed only the team’s indexes, and Collectord sends the team’s data through it. Set it up once per team:

1. In Splunk, create the team’s events index (team_a) and, if the cluster’s stats go to a metrics index, the team’s metrics index (team_a_metrics). Create a HEC token for the team whose default index is team_a and whose allowed indexes are those two.
2. Give the team an output on that token that names the team’s metrics index and drops what the token refuses. Either a SplunkOutput in the team’s namespace:

 1apiVersion: "collectord.io/v1"
 2kind: SplunkOutput
 3metadata:
 4  name: team
 5  namespace: team-a
 6spec:
 7  tokenFromSecret:
 8    secret: team-a-hec
 9    key: token
10  defaultMetricsIndex: team_a_metrics
11  incorrectIndexBehavior: Drop

Or a named output in the configuration (with Helm, under collectord.configuration.outputs.splunk.team-a):

1[output.splunk::team-a]
2token = <the team's HEC token>
3defaultMetricsIndex = team_a_metrics
4incorrectIndexBehavior = Drop

3. Annotate the team’s namespace with its output (splunk::team-a for the named output) and its events index:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: team-a
5  annotations:
6    collectord.io/output: splunk::user/team-a/team
7    collectord.io/index: team_a

4. In Splunk, give the team’s role search access to team_a and team_a_metrics only. A role that imports user also searches every index user may.

Collectord then sends what the namespace’s workloads produce through the team’s output: container and application logs, the namespace’s events and objects, pod, container, process and network stats, the socket table, and the pods’ Prometheus metrics. It all goes to team_a, except what goes to a metrics index, which goes to team_a_metrics.

Each setting closes one way out:

• collectord.io/index puts the team’s events index on every record. Without it, a record carries the index the cluster’s configuration names on its input, if any, and the team’s token refuses it.
• defaultMetricsIndex names the team’s metrics index; a token’s allowed indexes don’t choose one. Without it, the output takes [output.splunk]’s, the cluster’s metrics index, which the team’s token refuses. The metricsIndex and metricsOutput the cluster’s configuration sets on its inputs don’t apply to a team’s output.
• incorrectIndexBehavior: Drop drops a record the token refuses, such as one a pod’s own collectord.io/index sends to another index. The default, RedirectToDefault, sends it to the token’s default index instead: it stays in the team’s events index, where a metrics-index record is kept as the text metric.

Some data stays on the cluster’s outputs by design, even when it is about the team’s namespace:

• API server audit logs (the control plane’s host logs, which record the team’s requests too);
• cluster-scoped objects such as Nodes and PersistentVolumes;
• host logs and stats;
• Collectord’s own logs and metrics.

Know the limits:

• Annotations choose among the outputs that exist, and a pod’s annotation beats its namespace’s. Anyone who can annotate a pod can send its data through any named output in the configuration, the cluster’s main splunk output included. Keep the main output’s token off the teams’ indexes, so a team’s data reaches no other team’s index that way.
• Any pod can name an [output.splunk::<name>], and so write with that output’s token into the indexes it allows. A pod can name only a SplunkOutput of its own namespace (unless crossNamespaceUserOutputs is on), so prefer SplunkOutput where one team must not be able to write into another’s indexes.
• An output Collectord cannot use (a misspelt name, a deleted SplunkOutput or Secret, a setting it refuses) falls back to the main output until it can be used, and Collectord logs a warning. Until then the team’s data goes to the main output’s indexes.
• A Configuration resource that names an output reaches pods only: the namespace’s events and objects follow the namespace’s own annotations.

Logs sampling

Example 1. Random based sampling

When a container produces tens of thousands of lines per second and you only need to spot trends, such as error rates and latency distributions, forwarding the full volume is wasteful. collectord.io/logs-sampling-percent keeps a random percentage of the lines and drops the rest.

In the example below the application produces 300,000 lines; about 60,000 reach Splunk:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: logtest
 5  annotations:
 6    collectord.io/logs-sampling-percent: '20'
 7spec:
 8  containers:
 9  - name: logtest
10    image: docker.io/mffiedler/ocp-logtest:latest
11    args: [python, ocp_logtest.py,
12           --line-length=1024, --num-lines=300000, --rate=60000, --fixed-line]

Example 2. Hash-based sampling

Random sampling breaks per-user investigation: it can keep half of a user’s events and lose the other half, which makes correlation impossible. Hash-based sampling fixes this. Define a key, a named regex group such as a user ID or an IP, and Collectord either keeps every event with that key or drops them all.

The pod below samples by client IP:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-sampling
 5  annotations:
 6    collectord.io/logs-sampling-percent: '20'
 7    collectord.io/logs-sampling-key: '^(?P<key>(\d+\.){3}\d+)'
 8spec:
 9  containers:
10  - name: nginx-sampling
11    image: nginxinc/nginx-unprivileged

Thruput

Throttle a chatty container that would otherwise overwhelm the HEC pipeline and starve every other workload on the node. collectord.io/logs-ThruputPerSecond caps how many bytes of logs Collectord forwards for that container per second.

When the container goes over the limit, Collectord pauses reading and catches up once the rate drops, so events are delayed, not dropped. The backlog stays in the log file on disk. A container that stays over the limit long enough can lose events to log rotation before Collectord catches up.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-sampling
 5  annotations:
 6    collectord.io/logs-ThruputPerSecond: 128Kb
 7spec:
 8  containers:
 9  - name: nginx-sampling
10    image: nginxinc/nginx-unprivileged

To let the lines that matter through every cap, audit records for example, add a [pipe.thruput_exempt::<name>] section to the Collectord configuration (collectord.configuration.pipeThruputExempt in the Helm values). Its patternRegex selects the lines, and matchRegex.openshift_namespace the workloads. A workload it selects keeps reading over its cap instead of pausing, so its other lines over the cap are dropped, not delayed.

Time correction

When you start Collectord on a node that already has a long history of logs on disk, you usually don’t want last week’s logs in Splunk. Or you want to skip the future-dated noise from a misconfigured container. collectord.io/logs-TooOldEvents and collectord.io/logs-TooNewEvents define a window around “now”, and Collectord ignores events outside it.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-sampling
 5  annotations:
 6    collectord.io/logs-TooOldEvents: 168h
 7    collectord.io/logs-TooNewEvents: 1h
 8spec:
 9  containers:
10  - name: nginx-sampling
11    image: nginxinc/nginx-unprivileged

Handling multiple containers

A pod can have several containers, a web container and a user sidecar for example, and each often needs its own annotations. Prefix the annotation with the container name and a double dash: collectord.io/{container_name}--{annotation}. Annotations without a prefix apply to every container in the pod.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: webportal
 5  annotations:
 6    collectord.io/web--logs-index: 'web'
 7    collectord.io/web--logs-replace.2-search: '(?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}'
 8    collectord.io/web--logs-replace.2-val: '${IPv4p1}.X.X.X'
 9    collectord.io/user--logs-disabled: 'true'
10spec:
11  containers:
12  - name: web
13    image: nginxinc/nginx-unprivileged
14  - name: user
15    image: busybox
16    args: [/bin/sh, -c,
17           'while true; do wget -qO- localhost:8080 &> /dev/null; sleep 5; done']

Cluster level annotations

Sometimes one annotation should apply to every container that matches some criteria: every nginx image, every container in a label-selected namespace. Putting the annotation on every workload doesn’t scale. The Configuration CRD lets the platform team define such rules centrally, on the metadata fields Collectord already knows.

 1apiVersion: "collectord.io/v1"
 2kind: Configuration
 3metadata:
 4  name: apply-to-all-nginx
 5  annotations:
 6    collectord.io/nginx--logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
 7    collectord.io/nginx--logs-replace.1-val: ''
 8    collectord.io/nginx--logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
 9    collectord.io/nginx--logs-hashing.1-function: 'fnv-1a-64'
10spec:
11  openshift_container_image: "^(.*/)?nginx[^/]*$"

This applies the replace and hashing pipes to every container whose image name starts with nginx, whatever registry or organization it comes from: docker.io/library/nginx:latest, docker.io/nginxinc/nginx-unprivileged:1.27, and so on.

Collectord matches against the image name CRI-O reports, which is fully qualified: a pod with image: nginx reports docker.io/library/nginx:latest. That’s why the pattern allows a path before nginx. A pattern anchored as ^nginx matches nothing.

In the spec, you can match on any meta field Collectord forwards: openshift_container_image, openshift_container_name, openshift_daemonset_name, openshift_namespace, openshift_pod_labels, openshift_pod_name, and others. When you specify several fields, all of their regexes must match (logical AND).

Collectord doesn’t set the kubernetes_* names on OpenShift, so a Configuration copied from a Kubernetes cluster matches nothing until you rename its spec keys.

Forcing Cluster Level Annotations

By default, annotations on a Namespace, Deployment, or Pod beat anything coming from a cluster-level Configuration, which is what app teams want most of the time. When the platform team needs to enforce a policy, such as mandatory PII masking or a required index for compliance, the Configuration must win. Set force: true:

1apiVersion: "collectord.io/v1"
2kind: Configuration
3metadata:
4  name: apply-to-all-nginx
5  annotations:
6    collectord.io/index: bar
7spec:
8  openshift_container_image: "^(.*/)?nginx[^/]*$"
9force: true

Even with force: true, a more specific annotation still wins over a less specific one. collectord.io/logs-index=foo on a namespace beats a forced collectord.io/index=bar from a Configuration, because logs-index is type-specific.

Troubleshooting

When an annotation isn’t doing what you expect, check the Collectord logs for parser warnings. A typo in an annotation name shows up as:

1WARN 2018/08/31 21:05:33.122978 core/input/annotations.go:76: invalid annotation ...

Pipes that work on event data, such as field extraction and time parsing, report per-event errors in the collectord_errors field. To find the events that failed processing, search for collectord_errors=* in the affected index.

Describe command

collectord describe is the fastest way to confirm which annotations apply to a pod or container, and which level each came from. See Troubleshooting -> Describe.

Reduce log volume

Three sets of annotations act on a workload’s log volume:

• logs-repeat-* sends its repeated lines as one event that counts them;
• logs-fold-* folds storms of alike lines;
• logs-patterns-* reports its heaviest log patterns.

See Reduce log volume and the reference.

Reference

For the full list of every annotation grouped by datatype, see Annotations reference.