Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Kubernetes

Annotations

You can define annotations for namespaces, workloads, and pods. Annotations allow you to change how collectord forwards data to Splunk. Annotations also help collectord discover where the application logs are located.

The complete list of all available annotations is available at the bottom of this page.

Starting from version 5.9 you can define a subdomain for annotations under [general]annotationsSubdomain, that allows you to deploy multiple collectord instances and configure them with different annotations. If you want to use an annotation that applies to all collectord instance you can use a subdomain collectord.collectord.io/{annotation}={value}

Overriding indexes

Using annotations you can override to which index data should be forwarded from a specific namespace, workload, or pod. You can define one index for the whole object with collectord.io/index or specific indices for container logs collectord.io/logs-index, container stats collectord.io/stats-index, process stats collectord.io/procstats-index, and events collectord.io/events-index (can be applied only to the whole namespace).

As an example, if you want to override indexes for a specific namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: team1
  annotations:
    collectord.io/index: kubernetes_team1

This annotation tells collectord to forward all the data from this namespace to an index named kubernetes_team1. That includes pod and collectord stats, logs, and events.

When you change annotations for existing objects - it can take up to 2x[general.kubernetes]/timeout (2x5m by default) for that to take effect. That is how often collectord reloads metadata for all already monitored pods and compares the differences. You can always force this effect by recreating the monitored pod (wait [general.kubernetes]/metadataTTL after applying the change) or restarting collectord.

To define that you want to forward stats from Pods, Containers, and Processes to a dedicated index, you can define it as:

apiVersion: v1
kind: Namespace
metadata:
  name: team1
  annotations:
    collectord.io/index: kubernetes_team1
    collectord.io/procstats-index: kubernetes_team1_stats
    collectord.io/netstats-index: kubernetes_team1_stats
    collectord.io/nettable-index: kubernetes_team1_stats
    collectord.io/stats-index: kubernetes_team1_stats

That tells collectord that all the data from the namespace team1 should be forwarded to index kubernetes_team1, and pod, container (collectord.io/stats-index), and process stats (collectord.io/procstats-index) should be forwarded to index kubernetes_team1_stats. That means that events, container logs, and application logs will be forwarded to the kubernetes_team1 index.

collectord.io/logs-index overrides only the index for container logs. If you want to override the index for application logs, you should use collectord.io/index or collectord.io/volume.{N}-logs-index.

Similarly, you can override source, type and host.

Overriding index, source and type for specific events

Available since version 5.2

In cases when your container is running multiple processes, sometimes you want to override source or type just for specific events in the container (or application) logs. You can do that with override annotations.

For example, we will use the nginx image with logs:

172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

If we want to override the source of the web logs and keep all other logs with the predefined source:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-override.1-match: ^(\d{1,3}\.){3}\d{1,3}
    collectord.io/logs-override.1-source: /kubernetes/nginx/web-log
spec:
  containers:
  - name: nginx
    image: nginx

Collectord will override the source for matched events, so with our example you will end up with events similar to:

source                     | event
------------------------------------------------------------------------------------------------------------------------
/kubernetes/nginx/web-log  | 172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
/kubernetes/550...stderr   | 2018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
/kubernetes/nginx/web-log  | 172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

Replace patterns in events

You can define replace patterns with annotations. That allows you to hide sensitive information or drop unimportant information from the messages.

Replace patterns for container logs are configured with a pair of annotations grouped with the same number: collectord.io/logs-replace.1-search and collectord.io/logs-replace.1-val. The first specifies the search pattern as a regular expression, the second specifies a replace pattern. In replace patterns you can use placeholders for matches, like $1 or $name for named patterns.

We are using the Go regular expression library for replace pipes. You can find more information about the syntax at Package regexp and re2 syntax. We recommend using https://regex101.com for testing your patterns (set the Flavor to golang).

Using nginx as an example, our logs have a default pattern like:

172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 1. Replacing IPv4 addresses with X.X.X.X

If we want to hide IP addresses from the logs by replacing all IPv4 addresses with X.X.X.X:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-replace.1-search: (\d{1,3}\.){3}\d{1,3}
    collectord.io/logs-replace.1-val: X.X.X.X
spec:
  containers:
  - name: nginx
    image: nginx

The result of this replace pattern will be in Splunk:

X.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

You can also keep the first part of the IPv4 address with:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-replace.1-search: (?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}
    collectord.io/logs-replace.1-val: ${IPv4p1}.X.X.X
spec:
  containers:
  - name: nginx
    image: nginx

That results in:

172.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 2. Dropping messages

With replace patterns you can drop messages that you don’t want to see in Splunk. With the example below, we drop all log messages resulting from GET requests with a 200 response:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
    collectord.io/logs-replace.1-val: ''
    collectord.io/logs-replace.2-search: '(\d{1,3}\.){3}\d{1,3}'
    collectord.io/logs-replace.2-val: 'X.X.X.X'
spec:
  containers:
  - name: nginx
    image: nginx

In this example, we have two replace pipes. They apply in alphabetical order (replace.1 comes first, before replace.2).

X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 3. Whitelisting the messages

With the whitelist annotation, you can configure a pattern for the log messages, and only messages that match this pattern will be forwarded to Splunk:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-whitelist: '((DELETE)|(POST))$'
spec:
  containers:
  - name: nginx
    image: nginx

Hashing values in logs

Available since version 5.3

To hide sensitive data, you can use either replace patterns or hashing functions.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
    collectord.io/logs-hashing.1-function: 'fnv-1a-64'
spec:
  containers:
  - name: nginx
    image: nginx

This example will replace values that look like IP addresses in the string:

172.17.0.1 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

With the hashed value, in our example using the algorithm fnv-1a-64:

gqsxydjtZL4 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

Collectord supports a variety of hashing functions, including cryptographic hashing functions. The list of supported functions and their performance is listed below (performance in nanoseconds per operation to hash two IP addresses in a string source: 127.0.0.1, destination: 10.10.1.99):

| Function          | ns / op |
-------------------------------
| adler-32          |    1713 |
| crc-32-ieee       |    1807 |
| crc-32-castagnoli |    1758 |
| crc-32-koopman    |    1753 |
| crc-64-iso        |    1739 |
| crc-64-ecma       |    1740 |
| fnv-1-64          |    1711 |
| fnv-1a-64         |    1711 |
| fnv-1-32          |    1744 |
| fnv-1a-32         |    1738 |
| fnv-1-128         |    1852 |
| fnv-1a-128        |    1836 |
| md5               |    2032 |
| sha1              |    2037 |
| sha256            |    2220 |
| sha384            |    2432 |
| sha512            |    2516 |

Escaping terminal sequences, including terminal colors

Some containers do not turn off terminal colors automatically when they run inside a container. For example, if you run a container with an attached tty and define that you want to see colors:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-shell
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    tty: true
    command: [/bin/sh, -c,
             'while true; do ls --color=auto /; sleep 5; done;']

You can find messages similar to the following in Splunk:

[01;34mboot  etc  lib   media  opt  root  sbin  sys  usr
[0mbin   dev  home  lib64  mnt  proc  run   srv  tmp  var

You can easily escape them with the annotation collectord.io/logs-escapeterminalsequences='true':

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-shell
  annotations:
    collectord.io/logs-escapeterminalsequences: 'true'
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    tty: true
    command: [/bin/sh, -c,
             'while true; do ls --color=auto /; sleep 5; done;']

That way you will see logs in Splunk as you would expect:

bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

In the collectord configuration file, you can find [input.files]/stripTerminalEscapeSequencesRegex and [input.files]/stripTerminalEscapeSequences that define the default regexp used for removing terminal escape sequences and the default value for whether collectord should strip terminal escape sequences (defaults to false).

Extracting fields from the container logs

You can use field extraction, which allows you to extract timestamps from the messages and extract fields that will be indexed with Splunk to speed up searches.

Using the same example with nginx, we can define field extraction for some of the fields.

172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Important note: the first unnamed pattern is used as the message for the event. If you want to override it, you can use the annotation (available in 5.18+) collectord.io/logs-extractionMessageField to use a specific pattern as the message field.

Example 1. Extracting the timestamp

Assuming we want to keep the whole message as is and extract just a timestamp. We can define the extraction pattern with the regexp, specify that the timestampfield is timestamp, and define the timestampformat.

We use Go time parsing library, that defines the format with the specific date Mon Jan 2 15:04:05 MST 2006. See Go documentation for details.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-extraction: '^(.*\[(?P<timestamp>[^\]]+)\].+)$'
    collectord.io/logs-timestampfield: timestamp
    collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
spec:
  containers:
  - name: nginx
    image: nginx

This way, you will get messages in Splunk with the exact timestamp as specified in your container logs.

Collectord 5.24.440+: If the timestamp is in unixtime format, you can use format @unixtimestamp

Example 2. Extracting the fields

If you want to extract some fields and keep the message shorter, for example, if you have extracted the timestamps, there is no need to keep the timestamp in the raw message. In the example below, we extract the ip_address as a field, the timestamp, and keep the rest as a raw message.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-extraction: '^(?P<ip_address>[^\s]+) .* \[(?P<timestamp>[^\]]+)\] (.+)$'
    collectord.io/logs-timestampfield: timestamp
    collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
spec:
  containers:
  - name: nginx
    image: nginx

That results in messages:

ip_address | _time               | _raw
-----------|---------------------|-------------------------------------------------
172.17.0.1 | 2018-08-31 21:11:26 | "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 | 2018-08-31 21:11:32 | "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 | 2018-08-31 21:11:35 | "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Defining Event pattern

With the annotation collectord.io/logs-eventpattern, you can define how collectord should identify new events in the pipe. The default event pattern is defined by the collectord configuration as ^[^\s] (anything that does not start with a space character).

The default pattern works in most cases, but does not work in some, like Java exceptions, where the call stack of the error starts on the next line, and it does not start with a space character.

In the example below, we intentionally made a mistake in the configuration for ElasticSearch (s-node should be single-node) to get the error message:

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch-pod
spec:
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
    env:
    - name: discovery.type
      value: s-node

Results in

[2018-08-31T22:44:56,433][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/92] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
	at org.elasticsearch.discovery.DiscoveryModule.<init>(DiscoveryModule.java:129) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:477) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
	... 6 more
[2018-08-31T22:44:56,892][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

And with the default pattern, we will not have the warning line [2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] with the whole callstack.

We can define that every log event in this container should start with the [ character with the regular expression as:

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch-pod
  annotations:
    collectord.io/logs-eventpattern: '^\['
spec:
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
    env:
    - name: discovery.type
      value: s-node

Note: by default, collectord joins multi-line log lines that are written within a duration of 100ms, waits a maximum of 1s for the next line, and combines events up to a total of 100Kb. If you see that not all multi-line log lines are joined into one event as you expect, you might need to change the configuration for collectord under [pipe.join].

Application Logs

Complete guide for forwarding application logs from Kubernetes and OpenShift environments to Splunk

Sometimes it is hard or just not practical to redirect all logs from the container to stdout and stderr of the container. In such cases, you keep the logs in the container. We call them application logs. With collectord, you can easily pick up these logs and forward them to Splunk. No additional sidecars or processes are required inside your container.

Let’s take a look at the example below. We have a postgresql container that redirects most of the logs to a path inside the container /var/log/postgresql. We define for this container a volume (emptyDir driver) with the name psql_logs and mount it to /var/log/postgresql/. With the annotation collectord.io/volume.1-logs-name=psql_logs, we tell collectord to pick up all the logs with the default glob pattern *.log* (the default glob pattern is set in the collectord configuration, and you can override it with the annotation collectord.io/volume.{N}-logs-glob) in the volume and forward them automatically to Splunk.

When you need to forward logs from multiple volumes of the same container, you can group the settings with the same number, for example: collectord.io/volume.1-logs-name=psql_logs and collectord.io/volume.2-logs-name=psql_logs.

Example 1. Forwarding application logs

apiVersion: v1
kind: Pod
metadata:
  name: postgres-pod
  annotations:
    collectord.io/volume.1-logs-name: 'logs'
spec:
  containers:
  - name: postgres
    image: postgres
    command:
      - docker-entrypoint.sh
    args:
      - postgres
      - -c
      - logging_collector=on
      - -c
      - log_min_duration_statement=0
      - -c
      - log_directory=/var/log/postgresql
      - -c
      - log_min_messages=INFO
      - -c
      - log_rotation_age=1d
      - -c
      - log_rotation_size=10MB
    volumeMounts:
      - name: data
        mountPath: /var/lib/postgresql/data
      - name: logs
        mountPath: /var/log/postgresql/
  volumes:
  - name: data
    emptyDir: {}
  - name: logs
    emptyDir: {}

In the example above, the logs from the container will have a source similar to psql_logs:postgresql-2018-08-31_232946.log.

2018-08-31 23:31:02.034 UTC [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
	  c.relname as "Name",
	  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
	  pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
	FROM pg_catalog.pg_class c
	     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
	WHERE c.relkind IN ('r','p','')
	      AND n.nspname <> 'pg_catalog'
	      AND n.nspname <> 'information_schema'
	      AND n.nspname !~ '^pg_toast'
	  AND pg_catalog.pg_table_is_visible(c.oid)
	ORDER BY 1,2;
2018-08-31 23:30:53.490 UTC [124] FATAL:  role "postgresql" does not exist

Example 2. Forwarding application logs with fields extraction and time parsing

With annotations for application logs, you can define field extraction, replace patterns, and override the indexes, sources, and hosts.

As an example, with the extraction pattern and timestamp parsing, you can do:

apiVersion: v1
kind: Pod
metadata:
  name: postgres-pod
  annotations:
    collectord.io/volume.1-logs-name: 'logs'
    collectord.io/volume.1-logs-extraction: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} [^\s]+) (.+)$'
    collectord.io/volume.1-logs-timestampfield: 'timestamp'
    collectord.io/volume.1-logs-timestampformat: '2006-01-02 15:04:05.000 MST'
spec:
  containers:
  - name: postgres
    image: postgres
    command:
      - docker-entrypoint.sh
    args:
      - postgres
      - -c
      - logging_collector=on
      - -c
      - log_min_duration_statement=0
      - -c
      - log_directory=/var/log/postgresql
      - -c
      - log_min_messages=INFO
      - -c
      - log_rotation_age=1d
      - -c
      - log_rotation_size=10MB
    volumeMounts:
      - name: data
        mountPath: /var/lib/postgresql/data
      - name: logs
        mountPath: /var/log/postgresql/
  volumes:
  - name: data
    emptyDir: {}
  - name: logs
    emptyDir: {}

This way, you will extract the timestamps and remove them from the _raw message:

_time               | _raw
2018-08-31 23:31:02 | [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
                    | 	  c.relname as "Name",
                    | 	  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
                    | 	  pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
                    | 	FROM pg_catalog.pg_class c
                    | 	     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    | 	WHERE c.relkind IN ('r','p','')
                    | 	      AND n.nspname <> 'pg_catalog'
                    | 	      AND n.nspname <> 'information_schema'
                    | 	      AND n.nspname !~ '^pg_toast'
                    | 	  AND pg_catalog.pg_table_is_visible(c.oid)
                    | 	ORDER BY 1,2;
2018-08-31 23:30:53 |  UTC [124] FATAL:  role "postgresql" does not exist

Placeholder templates in a glob pattern

Available since Monitoring Kubernetes 5.20

In cases where you are mounting the same volume to multiple Pods and you want to differentiate the logs, you can now specify placeholders in the glob configuration. For example, if you have a volume mounted to a Pod with the name my-pod and to a Pod with the name my-pod-2, you can specify the glob configuration like this: {{kubernetes_pod_name}}.log, so Collectord will be able to identify that files my-pod.log and my-pod-2.log are coming from different Pods.

On Volume Database for acknowledgements

Available since Monitoring Kubernetes 5.20

Collectord has a database that stores information about files that were already processed. The database is stored by default on the host where Collectord is running. In case one of the volumes will be used on one host and then be mounted to another host, Collectord will start to process the files from the beginning. To avoid this, you can specify the annotation collectord.io/volume.{N}-logs-onvolumedatabase=true to enable the on-volume database. In this case, Collectord creates a database in the volume root .collectord.db, so the data about processed files will be stored in the volume and will be available on any host.

Important details about this feature: you need to mount the /rootfs directory in the Collectord container with write access. By default, the /rootfs directory is mounted as read-only.

Volume types

Collectord supports three volume types for application logs: emptyDir, hostPath, and persistentVolumeClaim. Collectord configuration has two settings that help collectord autodiscover application logs. The first is [general.kubernetes]/volumesRootDir for discovering volumes created with emptyDir, the second is [input.app_logs]/root for discovering host mounts, considering that they will be mounted with a different path to collectord.

Forwarding Prometheus metrics

Available since Monitoring Kubernetes 5.1.

You can define one or multiple ports that collectord should use to collect metrics in Prometheus format and forward to Splunk. The Collectord addon, which we deploy as part of the collectorforkubernetes.yaml, has a stanza that defines pod metrics autodiscovery [input.prometheus_auto]. The stanza determines the default interval, type, and timeout for the HTTP client.

The addon watches for new pods in the cluster, and when it sees annotations that can define a Prometheus endpoint, it sets up the collection right away. At minimum, you need to configure a port, for example, collectord.io/prometheus.1-port=9888.

For example, we use the sophos/nginx-prometheus-metrics image, which has a built-in plugin for nginx that can export metrics in Prometheus format. For that, we define the port that the addon needs to use to collect this data and the path.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: MyApp
  annotations:
    collectord.io/prometheus.1-port: '9527'
    collectord.io/prometheus.1-path: '/metrics'
spec:
  containers:
  - name: nginx
    image: sophos/nginx-prometheus-metrics

For details about metrics format, please read our documentation on Prometheus metrics.

Learn in the reference below of other available annotations for the Prometheus metrics.

Forwarding Prometheus metrics in Splunk Metrics Index

Available since Monitoring OpenShift 5.24

Collectord allows you to forward Prometheus metrics to Splunk in the Splunk Metrics Index. You can do this by setting annotations to define the index type as metrics and the index name. Make sure that the index is created in Splunk before you start forwarding the data and that the index type is set to metrics.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: MyApp
  annotations:
    collectord.io/prometheus.1-port: '9527'
    collectord.io/prometheus.1-path: '/metrics'
    collectord.io/prometheus.1-index: 'os_metrics'
    collectord.io/prometheus.1-output: 'splunk::metrics'
    collectord.io/prometheus.1-indexType: 'metrics'
spec:
  containers:
  - name: nginx
    image: sophos/nginx-prometheus-metrics

When you decide to configure Prometheus metrics to be sent to the Splunk Metrics Index, we suggest defining an additional Splunk Output that will forward to a Token that is set by default to use the Metrics Indexes (by default and additional indexes).

Change output destination

By default, collectord forwards all data to Splunk. You can configure containers to redirect data to devnull instead with the annotation collectord.io/output=devnull. This annotation changes the output for all data related to this container. Alternatively, you can change the output only for logs with the annotation collectord.io/logs-output=devnull.

By changing the default output for specific data, you can change how you forward data to Splunk. Instead of forwarding all logs by default, you can change the configuration for collectord with --env "COLLECTOR__LOGS_OUTPUT=input.files__output=devnull" to specify not to forward container logs by default. Then define with containers which logs you want to see in Splunk with collectord.io/logs-output=splunk.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: MyApp
  annotations:
    collectord.io/logs-output: 'splunk'
spec:
  containers:
  - name: nginx
    image: nginx

Additionally, if you configured multiple Splunk outputs with the configuration (see Support for multiple Splunk clusters) you can forward the data to a specific Splunk Cluster as

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: MyApp
  annotations:
    collectord.io/output: 'splunk::prod1'
spec:
  containers:
  - name: nginx
    image: nginx

Forwarding logs to multiple Splunk HTTP Event Collector endpoints simultaneously

Available since Monitoring Kubernetes 5.20

With annotation collectord.io/logs-output you can configure multiple Splunk HEC endpoints, for example splunk::apps and splunk::security using the comma separated list: collectord.io/logs-output=splunk::apps,splunk::security. Assuming you have them defined in the ConfigMap like [output.splunk::apps] and [output.splunk::security].

Additionally, you can configure indexes for the endpoints in square brackets, for example collectord.io/logs-output=splunk::apps[kubernetes_logs],splunk::security[kubernetes_security].

In that case, each event will be sent to both Splunk HEC endpoints.

For example, if you want to forward logs from a specific container to multiple Splunk HEC endpoints, you can use the following

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: MyApp
  annotations:
    collectord.io/logs-output: 'splunk::apps[kubernetes_logs],splunk::security[kubernetes_security]'
spec:
  containers:
  - name: nginx
    image: nginx

User outputs

Available since Monitoring Kubernetes 5.22

User can define a Splunk output with the CustomResourceDefinition (CRD) SplunkOutput in their namespace. For example

apiVersion: "collectord.io/v1"
kind: SplunkOutput
metadata:
  name: splunk-user-output-for-deployment
spec:
  token: 1a8b9c3e-7789-4353-821f-15b9662bac99
  url: https://splunk.example.com:8088/services/collector/event/1.0
  insecure: true

Similarly, to how you can reference the default Splunk outputs defined in the ConfigMap, you can reference them with annotation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: long-running
  annotations:
    collectord.io/output: splunk::user/default/splunk-user-output-for-deployment
spec:
  ...

Where you define it as splunk::user/<namespace>/<name>.

Logs sampling

Available since collectorforkubernetes v5.6

Example 1. Random based sampling

When the application produces a high amount of logs, in some cases it could be enough to just look at the sampled amount of the logs to understand how many failed requests the application has, or how it behaves. You can add an annotation for the logs to specify the percent amount of the logs that should be forwarded to Splunk.

In the following example, this application produces 300,000 log lines. Only about 60,000 log lines are going to be forwarded to Splunk.

apiVersion: v1
kind: Pod
metadata:
  name: logtest
  annotations:
    collectord.io/logs-sampling-percent: '20'
spec:
  restartPolicy: Never
  containers:
  - name: logtest
    image: docker.io/mffiedler/ocp-logtest:latest
    args: [python, ocp_logtest.py,
           --line-length=1024, --num-lines=300000, --rate=60000, --fixed-line]

Example 2. Hash-based sampling

In situations where you want to look at the pattern for a specific user, you can specify that you want to sample logs based on the hash value, to be sure that if the same key is present in two different log lines, both of them will be forwarded to Splunk.

In the following example we define a key (should be a named submatch pattern) as an IP address.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-sampling
  annotations:
    collectord.io/logs-sampling-percent: '20'
    collectord.io/logs-sampling-key: '^(?P<key>(\d+\.){3}\d+)'
spec:
  containers:
  - name: nginx-sampling
    image: nginx

Thruput

Available since collectorforkubernetes v5.10.252

You can configure the thruput specifically for the container logs as

apiVersion: v1
kind: Pod
metadata:
  name: nginx-sampling
  annotations:
    collectord.io/logs-ThruputPerSecond: 128Kb
spec:
  containers:
  - name: nginx-sampling
    image: nginx

In case this container produces more than 128kb per second collectord will throttle logs.

Time correction

Available since collectorforkubernetes v5.10.252

If you are pre-loading a lot of logs, you might want to configure the events that you want to skip, as they are too old or too new

apiVersion: v1
kind: Pod
metadata:
  name: nginx-sampling
  annotations:
    collectord.io/logs-TooOldEvents: 168h
    collectord.io/logs-TooNewEvents: 1h
spec:
  containers:
  - name: nginx-sampling
    image: nginx

Handling multiple containers

A pod can have multiple containers. You can define annotations for a specific container with the name prefixed to the annotation. The format of the annotations is collectord.io/{container_name}--{annotation}: {annotation-value}. As an example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/web--logs-index: 'web'
    collectord.io/web--logs-replace.2-search: '(?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}'
    collectord.io/web--logs-replace.2-val: '${IPv4p1}.X.X.X'
    collectord.io/user--logs-disabled: 'true'
spec:
  containers:
  - name: web
    image: nginx
  - name: user
    image: busybox
    args: [/bin/sh, -c,
           'while true; do wget -qO- localhost:80 &> /dev/null; sleep 5; done']

Cluster level annotations

Available since collectorforkubernetes v5.12.270

You can apply annotations to pods on the cluster level with Configuration in api group collectord.io/v1. For example:

apiVersion: "collectord.io/v1"
kind: Configuration
metadata:
  name: apply-to-all-nginx
  annotations:
    collectord.io/nginx--logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
    collectord.io/nginx--logs-replace.1-val: ''
    collectord.io/nginx--logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
    collectord.io/nginx--logs-hashing.1-function: 'fnv-1a-64'
spec:
  kubernetes_container_image: "^nginx(:.*)?$"

This configuration will be applied to all containers that use an image with nginx in the name (examples are nginx:latest or nginx:1.0).

In the spec of the Configuration you include selectors based on the meta fields that we forward to Splunk, which can include fields like kubernetes_container_image, kubernetes_container_name, kubernetes_daemonset_name, kubernetes_namespace, kubernetes_pod_labels, kubernetes_pod_name, etc. When you specify multiple fields in the spec, all regexes must match.

Forcing Cluster Level Annotations

Available since collectorforkubernetes v5.19.390

If you already have an annotation, for example, collectord.io/index=foo defined on a Namespace, Deployment or Pod, and you are trying to apply this annotation from Cluster Level Configuration as collectord.io/index=bar, the one from the objects will take priority.

With the force modifier you can override those annotations, even if you have them defined on the objects.

apiVersion: "collectord.io/v1"
kind: Configuration
metadata:
  name: apply-to-all-nginx
  annotations:
    collectord.io/index=bar
spec:
  kubernetes_container_image: "^nginx(:.*)?$"
force: true

NOTE: if you have an annotation defined in the namespace as collectord.io/logs-index=foo, it will still take priority over index=bar, as logs-index=foo is type specific.

Troubleshooting

Check the collectord logs for warning messages about the annotations, you can find if you made a misprint in the annotations if you see warnings like

WARN 2018/08/31 21:05:33.122978 core/input/annotations.go:76: invalid annotation ...

Some pipes, like fields extraction and time parsing pipes adds a error in the field collectord_errors, so you can identify when some events failed to be processed by this pipe.

Describe command

Please use Collectord describe command to see how annotations are applied to a specific pod or container.

See Troubleshooting -> Describe.

Reference

  • General annotations
    • collectord.io/index - change the index for all the data forwarded for this Pod (metrics, container logs, application logs)
    • collectord.io/source - change the source for all the data forwarded for this Pod (metrics, container logs, application logs)
    • collectord.io/type - change the sourcetype for all the data forwarded for this Pod (metrics, container logs, application logs)
    • collectord.io/host - change the host for all the data forwarded for this Pod (metrics, container logs, application logs)
    • collectord.io/output - (5.2+) change the output to devnull or splunk
    • collectord.io/userfields.{fieldname} - (5.15.300+) attach custom fields to events
  • Annotations for container logs
    • collectord.io/logs-index - change the index for the container logs forwarded from this Pod
    • collectord.io/logs-source - change the source for the container logs forwarded from this Pod
    • collectord.io/logs-type - change the sourcetype for the container logs forwarded from this Pod
    • collectord.io/logs-host - change the host for the container logs forwarded from this Pod
    • collectord.io/logs-eventpattern - set the regex identifying the event start pattern for Pod logs
    • collectord.io/logs-replace.{N}-search - define the search pattern for the replace pipe
    • collectord.io/logs-replace.{N}-val - define the replace pattern for the replace pipe
    • collectord.io/logs-hashing.{N}-match - (5.3+) the regexp for a matched value
    • collectord.io/logs-hashing.{N}-function - (5.3+) hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
    • collectord.io/logs-extraction - define the regexp for fields extraction
    • collectord.io/logs-extractionMessageField - (5.18+) specify the field name for the message (by default first unnamed ground in regexp)
    • collectord.io/logs-timestampfield - define the field for timestamp (after fields extraction)
    • collectord.io/logs-timestampformat - define the timestamp format
    • collectord.io/logs-timestampsetmonth - define if month should be set to current for timestamp
    • collectord.io/logs-timestampsetday - define if day should be set to current for timestamp
    • collectord.io/logs-timestamplocation - define timestamp location if not set by format
    • collectord.io/logs-joinpartial - join partial events
    • collectord.io/logs-joinmultiline - (5.3+) join multiline logs (default value depends on [pipe.join] disabled)
    • collectord.io/logs-escapeterminalsequences - escape terminal sequences (including colors)
    • collectord.io/logs-override.{N}-match - (5.2+) match for override pattern
    • collectord.io/logs-override.{N}-index - (5.2+) override index for matched events
    • collectord.io/logs-override.{N}-source - (5.2+) override source for matched events
    • collectord.io/logs-override.{N}-type - (5.2+) override type for matched events
    • collectord.io/logs-output - (5.2+) change the output to devnull or splunk (this annotation cannot be specified for stderr and stdout)
    • collectord.io/logs-disabled - (5.3+) disable any log processing for this container (this annotation cannot be specified for stderr and stdout)
    • collectord.io/logs-sampling-percent - (5.6+) specify the % value of logs that should be forwarded to splunk
    • collectord.io/logs-sampling-key - (5.6+) regexp pattern to specify the key for the sampling based on hash values
    • collectord.io/logs-ThruputPerSecond - (5.10.252+) set the thruput for this container, maximum amount of logs per second, for example 128Kb, 1024b
    • collectord.io/logs-TooOldEvents - (5.10.252+) duration of events from now to past that are considered too old and should be ignored, for example 168h, 24h
    • collectord.io/logs-TooNewEvents - (5.10.252+) duration of events from now to the future that are considered too new and should be ignored, for example 1h, 30m
    • collectord.io/logs-whitelist - (5.14.284+) allow to configure pattern for log messages, only log messages matching this pattern will be forwarded to Splunk
    • collectord.io/logs-userfields.{fieldname} - (5.15.300+) attach custom fields to events
    • Specific for stdout, with the annotations below you can define configuration specific for stdout
      • collectord.io/stdout-logs-index
      • collectord.io/stdout-logs-source
      • collectord.io/stdout-logs-type
      • collectord.io/stdout-logs-host
      • collectord.io/stdout-logs-eventpattern
      • collectord.io/stdout-logs-replace.{N}-search
      • collectord.io/stdout-logs-replace.{N}-val
      • collectord.io/stdout-logs-hashing.{N}-match - (5.3+)
      • collectord.io/stdout-logs-hashing.{N}-function - (5.3+)
      • collectord.io/stdout-logs-extraction
      • collectord.io/stdout-logs-extractionMessageField - (5.18+)
      • collectord.io/stdout-logs-timestampfield
      • collectord.io/stdout-logs-timestampformat
      • collectord.io/stdout-logs-timestampsetmonth
      • collectord.io/stdout-logs-timestampsetday
      • collectord.io/stdout-logs-timestamplocation
      • collectord.io/stdout-logs-joinpartial
      • collectord.io/stdout-logs-joinmultiline - (5.3+)
      • collectord.io/stdout-logs-escapeterminalsequences
      • collectord.io/stdout-logs-override.{N}-match - (5.2+)
      • collectord.io/stdout-logs-override.{N}-index - (5.2+)
      • collectord.io/stdout-logs-override.{N}-source - (5.2+)
      • collectord.io/stdout-logs-override.{N}-type - (5.2+)
      • collectord.io/stdout-logs-sampling-percent - (5.6+)
      • collectord.io/stdout-logs-sampling-key - (5.6+)
      • collectord.io/stdout-logs-ThruputPerSecond - (5.10.252+)
      • collectord.io/stdout-logs-TooOldEvents - (5.10.252+)
      • collectord.io/stdout-logs-TooNewEvents - (5.10.252+)
      • collectord.io/stdout-logs-whitelist - (5.14.284+)
    • Specific for stderr, with the annotations below you can define configuration specific for stderr
      • collectord.io/stderr-logs-index
      • collectord.io/stderr-logs-source
      • collectord.io/stderr-logs-type
      • collectord.io/stderr-logs-host
      • collectord.io/stderr-logs-eventpattern
      • collectord.io/stderr-logs-replace.{N}-search
      • collectord.io/stderr-logs-replace.{N}-val
      • collectord.io/stderr-logs-hashing.{N}-match - (5.3+)
      • collectord.io/stderr-logs-hashing.{N}-function - (5.3+)
      • collectord.io/stderr-logs-extraction
      • collectord.io/stdout-logs-extractionMessageField - (5.18+)
      • collectord.io/stderr-logs-timestampfield
      • collectord.io/stderr-logs-timestampformat
      • collectord.io/stderr-logs-timestampsetmonth
      • collectord.io/stderr-logs-timestampsetday
      • collectord.io/stderr-logs-timestamplocation
      • collectord.io/stderr-logs-joinpartial
      • collectord.io/stderr-logs-joinmultiline - (5.3+)
      • collectord.io/stderr-logs-escapeterminalsequences
      • collectord.io/stderr-logs-override.{N}-match - (5.2+)
      • collectord.io/stderr-logs-override.{N}-index - (5.2+)
      • collectord.io/stderr-logs-override.{N}-source - (5.2+)
      • collectord.io/stderr-logs-override.{N}-type - (5.2+)
      • collectord.io/stderr-logs-sampling-percent - (5.6+)
      • collectord.io/stderr-logs-sampling-key - (5.6+)
      • collectord.io/stderr-logs-ThruputPerSecond - (5.10.252+)
      • collectord.io/stderr-logs-TooOldEvents - (5.10.252+)
      • collectord.io/stderr-logs-TooNewEvents - (5.10.252+)
      • collectord.io/stderr-logs-whitelist - (5.14.284+)
  • Annotations for container stats
    • collectord.io/stats-index - change the index for the container metrics forwarded from this Pod
    • collectord.io/stats-source - change the source for the container metrics forwarded from this Pod
    • collectord.io/stats-type - change the sourcetype for the container metrics forwarded from this Pod
    • collectord.io/stats-host - change the host for the container metrics forwarded from this Pod
    • collectord.io/stats-output - (5.2+) change the output to devnull or splunk
    • collectord.io/stats-interval - (5.14+) override default interval how often to collect stats
    • collectord.io/stats-userfields.{fieldname} - (5.15.300+) attach custom fields to events
  • Annotations for container processes stats
    • collectord.io/procstats-index - change the index for the container process metrics forwarded from this Pod
    • collectord.io/procstats-source - change the source for the container process metrics forwarded from this Pod
    • collectord.io/procstats-type - change the type for the container process metrics forwarded from this Pod
    • collectord.io/procstats-host - change the host for the container process metrics forwarded from this Pod
    • collectord.io/procstats-output - (5.2+) change the output to devnull or splunk
    • collectord.io/procstats-userfields.{fieldname} - (5.15.300+) attach custom fields to events
    • collectord.io/procstats-hideargs - (5.24.440+) hide the command line arguments for the process
  • Annotations for container network stats
    • collectord.io/netstats-index - change the index for the container network metrics forwarded from this Pod
    • collectord.io/netstats-source - change the source for the container network metrics forwarded from this Pod
    • collectord.io/netstats-type - change the type for the container network metrics forwarded from this Pod
    • collectord.io/netstats-host - change the host for the container network metrics forwarded from this Pod
    • collectord.io/netstats-output - (5.2+) change the output to devnull or splunk
    • collectord.io/netstats-interval - (5.14+) override default interval how often to collect stats
    • collectord.io/netstats-userfields.{fieldname} - (5.15.300+) attach custom fields to events
  • Annotations for container network socket table
    • collectord.io/nettable-index - change the index for the container network socket table forwarded from this Pod
    • collectord.io/nettable-source - change the source for the container network socket table forwarded from this Pod
    • collectord.io/nettable-type - change the type for the container network socket table forwarded from this Pod
    • collectord.io/nettable-host - change the host for the container network socket table forwarded from this Pod
    • collectord.io/nettable-output - (5.2+) change the output to devnull or splunk
    • collectord.io/nettable-interval - (5.14+) override default interval how often to collect sta
    • collectord.io/nettable-userfields.{fieldname} - (5.15.300+) attach custom fields to events
  • Annotations for events (can be applied only to namespaces)
    • collectord.io/events-index - change the index for the events of specific namespace
    • collectord.io/events-source - change the source for the events of specific namespace
    • collectord.io/events-type - change the source type for the events of specific namespace
    • collectord.io/events-host - change the host for the events of specific namespace
    • collectord.io/events-userfields.{fieldname} - (5.15.300+) attach custom fields to events
  • Annotations for application logs
    • collectord.io/volume.{N}-logs-name - name of the volume attached to Pod
    • collectord.io/volume.{N}-logs-index - target index for logs forwarded from the volume
    • collectord.io/volume.{N}-logs-source - change the source for logs forwarded from the volume
    • collectord.io/volume.{N}-logs-type - change the type for logs forwarded from the volume
    • collectord.io/volume.{N}-logs-host - change the host for logs forwarded from the volume
    • collectord.io/volume.{N}-logs-eventpattern - change the event pattern defining new event for logs forwarded from the volume
    • collectord.io/volume.{N}-logs-replace.{N}-search - specify the regex search for replace pipe for the logs
    • collectord.io/volume.{N}-logs-replace.{N}-val - specify the regex replace pattern for replace pipe for the logs
    • collectord.io/volume.{N}-logs-hashing.{N}-match - (5.3+) the regexp for a matched value
    • collectord.io/volume.{N}-logs-hashing.{N}-function - (5.3+) hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
    • collectord.io/volume.{N}-logs-extraction - specify the fields extraction with the regex the logs
    • collectord.io/volume.{N}-logs-extractionMessageField - (5.18+) specify the field name for the message (by default first unnamed ground in regexp)
    • collectord.io/volume.{N}-logs-timestampfield - specify the timestamp field
    • collectord.io/volume.{N}-logs-timestampformat - specify the format for timestamp field
    • collectord.io/volume.{N}-logs-timestampsetmonth - define if month should be set to current for timestamp
    • collectord.io/volume.{N}-logs-timestampsetday - define if day should be set to current for timestamp
    • collectord.io/volume.{N}-logs-timestamplocation - define timestamp location if not set by format
    • collectord.io/volume.{N}-logs-glob - set the glob pattern for matching logs
    • collectord.io/volume.{N}-logs-match - set the regexp pattern for matching logs
    • collectord.io/volume.{N}-logs-recursive - set if walker should walk the directory recursive
    • collectord.io/volume.{N}-logs-override.{N}-match - (5.2+) match for override pattern
    • collectord.io/volume.{N}-logs-override.{N}-index - (5.2+) override index for matched events
    • collectord.io/volume.{N}-logs-override.{N}-source - (5.2+) override source for matched events
    • collectord.io/volume.{N}-logs-override.{N}-type - (5.2+) override type for matched events
    • collectord.io/volume.{N}-logs-sampling-percent - (5.6+) specify the % value of logs that should be forwarded to splunk
    • collectord.io/volume.{N}-logs-sampling-key - (5.6+) regexp pattern to specify the key for the sampling based on hash values
    • collectord.io/volume.{N}-logs-ThruputPerSecond - (5.10.252+) set the thruput for this container, maximum amount of logs per second, for example 128Kb, 1024b
    • collectord.io/volume.{N}-logs-TooOldEvents - (5.10.252+) duration of events from now to past that are considered too old and should be ignored, for example 168h, 24h
    • collectord.io/volume.{N}-logs-TooNewEvents - (5.10.252+) duration of events from now to the future that are considered too new and should be ignored, for example 1h, 30m
    • collectord.io/volume.{N}-logs-whitelist - (5.14.284+) allow to configure pattern for log messages, only log messages matching this pattern will be forwarded to Splunk
    • collectord.io/volume.{N}-logs-userfields.{fieldname} - (5.15.300+) attach custom fields to events
    • collectord.io/volume.{N}-logs-maxholdafterclose - (5.18.381+) how long Collectord can hold file descriptors open for files in PVC after pod is terminated (duration 5s, 1800s)
    • collectord.io/volume.{N}-logs-onvolumedatabase - (5.20.400+) boolean flag to enable on volume database for this volume, in case if this volume might be used on more than one host
    • collectord.io/volume.{N}-logs-withlock - (5.24.440+) boolean flag to enable file locking of forwarding logs from the files, and preventing other Collectord instances to read the file
  • Annotations for forwarding metrics in Prometheus format
    • collectord.io/prometheus.{N}-port - (required) specify the port that needs to be used to collect metrics
    • collectord.io/prometheus.{N}-index - specify target index for the metrics
    • collectord.io/prometheus.{N}-source - specify target source for the metrics (default is /kubernetes/{kubernetes_pod_id})
    • collectord.io/prometheus.{N}-type - specify the type for the metrics (default is set in [input.prometheus_auto] of collectord configuration)
    • collectord.io/prometheus.{N}-host - specify the host for the metrics (by default node name is used)
    • collectord.io/prometheus.{N}-path - specify the path in url for the metrics
    • collectord.io/prometheus.{N}-query - (5.17+) specify the raw query in url for the metrics
    • collectord.io/prometheus.{N}-interval - how often to collect metrics (default is set in [input.prometheus_auto] of collectord configuration)
    • collectord.io/prometheus.{N}-includehelp - include help for metrics (default is false)
    • collectord.io/prometheus.{N}-insecure - insecure if scheme is https (default is false)
    • collectord.io/prometheus.{N}-scheme - scheme (http or https, default is http)
    • collectord.io/prometheus.{N}-username - (5.2+) basic auth username
    • collectord.io/prometheus.{N}-password - (5.2+) basic auth password
    • collectord.io/prometheus.{N}-whitelist - (5.15.300+) whitelist regexp
    • collectord.io/prometheus.{N}-blacklist - (5.15.300+) blacklist regexp
    • collectord.io/prometheus.{N}-authorizationkey - (5.16.350+) include Authorization header with the value specified in the ConfigMap under [input.prometheus_auto]
    • collectord.io/prometheus.{N}-caname - (5.16.350+) is used to verify the hostname on the returned certificates unless insecure is set
    • collectord.io/prometheus.{N}-output - (5.23.430+) specify output
    • collectord.io/prometheus.{N}-indexType - (5.24.440+) specify the index type for the metrics (default is events), you can change it to metrics

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.