You can define annotations for namespaces, workloads, and pods. Annotations allow you to change how collectord forwards data to ElasticSearch. Annotations also help collectord where to discover the application logs.

The complete list of all the available annotations is available at the bottom of this page.

Default configuration uses annotationSubdomain = elasticsearch, so all the annotations should start with elasticsearch.collectord.io/. So ElasticSearch and ElasticSearch annotations are different.

Overriding datastreams

Using annotations, you can override to which datastream data should be forwarded from a specific namespace, workload, or pod. You can define one datastream for the whole object with elasticsearch.collectord.io/index or specific datastreams for container logs elasticsearch.collectord.io/logs-index, and events elasticsearch.collectord.io/events-index (can be applied only to whole namespace).

As an example, if you want to override indexes for a specific namespace

yaml

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: team1
5  annotations:
6    elasticsearch.collectord.io/index: logs-team1

This annotation tells collectord to forward all the data from this namespace to datastream named logs-team1.

elasticsearch.collectord.io/logs-index overrides only datastream for the container logs. If you want to override logs for the application logs you should use elasticsearch.collectord.io/index or elasticsearch.collectord.io/volume.{N}-logs-index.

Overriding datastream for specific events

In the case when your container is running multiple processes, sometimes you want to override datastream just for specific events in the container (or application) logs. You can do that with the override annotations.

For example, we will use the nginx image with logs

text

1172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
22018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
3172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

If we want to override datastream of the web logs and keep all other logs with the predefined datastream

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-override.1-match: ^(\d{1,3}\.){3}\d{1,3}
 7    elasticsearch.collectord.io/logs-override.1-index: logs-nginx-web
 8spec:
 9  containers:
10  - name: nginx
11    image: nginx

The collectord will override datastream for matched events, so with our example you will end up with events similar to

text

1datastream       | event
2------------------------------------------------------------------------------------------------------------------------
3logs-nginx-web   | 172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
4logs-collectord  | 2018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
5logs-nginx-web   | 172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

Replace patterns in events

You can define replace patterns with the annotations. That allows you to hide sensitive information or drop unimportant information from the messages.

Replace patterns for container logs are configured with a pair of annotations grouped with the same number elasticsearch.collectord.io/logs-replace.1-search and elasticsearch.collectord.io/logs-replace.2-val, first specifies the search pattern as a regular expression, second a replace pattern. In replace patterns you can use placeholders for matches, like $1 or $name for named patterns.

We’re using a Go regular expression library in implementation of replace pipes. You can find more information about the syntax at Package regexp and re2 syntax. We recommend to use https://regex101.com for testing your patterns (set the Flavor to golang).

Using nginx as an example, our logs have a default pattern like

text

1172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 1. Replacing IPv4 addresses with `X.X.X.X`

If we want to hide an IP address from the logs by replacing all IPv4 addresses with X.X.X.X

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-replace.1-search: (\d{1,3}\.){3}\d{1,3}
 7    elasticsearch.collectord.io/logs-replace.1-val: X.X.X.X
 8spec:
 9  containers:
10  - name: nginx
11    image: nginx

The result of this replace pattern will be in ElasticSearch

text

1X.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

You can also keep the first part of the IPv4 with

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-replace.1-search: (?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}
 7    elasticsearch.collectord.io/logs-replace.1-val: ${IPv4p1}.X.X.X
 8spec:
 9  containers:
10  - name: nginx
11    image: nginx

That results in

text

1172.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2172.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3172.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 2. Dropping messages

With the replace patterns, you can drop messages that you don’t want to see in ElasticSearch. With the example below we drop all log messages resulted from GET requests with 200 response

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
 7    elasticsearch.collectord.io/logs-replace.1-val: ''
 8    elasticsearch.collectord.io/logs-replace.2-search: '(\d{1,3}\.){3}\d{1,3}'
 9    elasticsearch.collectord.io/logs-replace.2-val: 'X.X.X.X'
10spec:
11  containers:
12  - name: nginx
13    image: nginx

In this example, we have two replace pipes. They apply in the alphabetical order (replace.1 comes first, before the replace.2).

text

1X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
2X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 3. Whitelisting the messages

With the whitelist annotation you can configure a pattern for the log messages, and only messages that match this pattern will be forwarded to ElasticSearch

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-whitelist: '((DELETE)|(POST))$'
 7spec:
 8  containers:
 9  - name: nginx
10    image: nginx

Hashing values in logs

To hide sensitive data, you can use replace patterns or hashing functions.

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
 7    elasticsearch.collectord.io/logs-hashing.1-function: 'fnv-1a-64'
 8spec:
 9  containers:
10  - name: nginx
11    image: nginx

This example will replace values that look like an IP address in the string

text

1172.17.0.1 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

With the hashed value, in our example using the algorithm fnv-1a-64

text

1gqsxydjtZL4 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

Collectord supports a variety of hashing functions, including cryptographic hashing functions. The list of supported functions and their performance is listed below (performance in the nanoseconds per operation to hash two IP addresses in a string source: 127.0.0.1, destination: 10.10.1.99)

text

 1| Function          | ns / op |
 2-------------------------------
 3| adler-32          |    1713 |
 4| crc-32-ieee       |    1807 |
 5| crc-32-castagnoli |    1758 |
 6| crc-32-koopman    |    1753 |
 7| crc-64-iso        |    1739 |
 8| crc-64-ecma       |    1740 |
 9| fnv-1-64          |    1711 |
10| fnv-1a-64         |    1711 |
11| fnv-1-32          |    1744 |
12| fnv-1a-32         |    1738 |
13| fnv-1-128         |    1852 |
14| fnv-1a-128        |    1836 |
15| md5               |    2032 |
16| sha1              |    2037 |
17| sha256            |    2220 |
18| sha384            |    2432 |
19| sha512            |    2516 |

Escaping terminal sequences, including terminal colors

Some containers don’t turn off terminal colors automatically when they run inside container. For example, if you run container with attached tty and define that you want to see colors

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: ubuntu-shell
 5spec:
 6  containers:
 7  - name: ubuntu
 8    image: ubuntu
 9    tty: true
10    command: [/bin/sh, -c,
11             'while true; do ls --color=auto /; sleep 5; done;']

You can find messages similar to below in ElasticSearch

text

1[01;34mboot[0m  [01;34metc[0m  [01;34mlib[0m   [01;34mmedia[0m  [01;34mopt[0m  [01;34mroot[0m  [01;34msbin[0m  [01;34msys[0m  [01;34musr[0m
2[0m[01;34mbin[0m   [01;34mdev[0m  [01;34mhome[0m  [01;34mlib64[0m  [01;34mmnt[0m  [01;34mproc[0m  [01;34mrun[0m   [01;34msrv[0m  [30;42mtmp[0m  [01;34mvar[0m

You can easily escape them with the annotation elasticsearch.collectord.io/logs-escapeterminalsequences='true'

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: ubuntu-shell
 5  annotations:
 6    elasticsearch.collectord.io/logs-escapeterminalsequences: 'true'
 7spec:
 8  containers:
 9  - name: ubuntu
10    image: ubuntu
11    tty: true
12    command: [/bin/sh, -c,
13             'while true; do ls --color=auto /; sleep 5; done;']

That way you will see logs in ElasticSearch as you would expect

text

1bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
2boot  etc  lib   media  opt  root  sbin  sys  usr

In the collectord configuration file you can find [input.files]/stripTerminalEscapeSequencesRegex and [input.files]/stripTerminalEscapeSequences that defines default regexp used for removing terminal escape sequences and default value if collectord should strip terminal escape sequences (defaults to false).

Extracting fields from the container logs

You can use fields extraction, which allows you to extract timestamps from the messages, extract fields that will be indexed with ElasticSearch to speed up the search.

Using the same example with nginx we can define fields extraction for some fields.

text

1172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
3172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Important note that the first unnamed pattern is used as the message for the event. If you want to override it, you can use the annotations elasticsearch.collectord.io/logs-extractionMessageField to use a specific pattern as a message field.

Nested Objects

Considering that you might want to create nested JSON objects in the events sent to ElasticSearch, when you add a named group with double underscore (__) in the name, the collectord will replace it with dot (.). For example, if you’ve named groups in the extraction pattern (?P<obj__firstname>value1)\w+ (?P<obj__lastname>value2)\w+, the collectord will create and object {"obj":{"firstname": "value", "lastname": "value"}}.

Datastreams and index templates

When you extract new fields, you might want to create a new index template for the new fields. Be sure to add elasticsearch.collectord.io/logs-datastream annotation to the pod, so the collectord will send the data to a new datastream.

Example 1. Extracting the timestamp

Assuming we want to keep the whole message as it is, and extract just a timestamp. We can define the extraction pattern with the regexp. Specify that the timestampfield is timestamp and define the timestampformat.

We use Go time parsing library, that defines the format with the specific date Mon Jan 2 15:04:05 MST 2006. See Go documentation for details.

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-extraction: '^(.*\[(?P<timestamp>[^\]]+)\].+)$'
 7    elasticsearch.collectord.io/logs-timestampfield: timestamp
 8    elasticsearch.collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
 9spec:
10  containers:
11  - name: nginx
12    image: nginx

In that way, you will get messages in ElasticSearch with the exact timestamp as specified in your container logs.

Example 2. Extracting the fields

If you want to extract some fields and keep the message shorter, as an example, if you’ve extracted the timestamps, there is no need for you to keep the timestamp in the raw message. In the example below we extract the ip_address address as a field, timestamp and keep the rest as a raw message.

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-extraction: '^(?P<ip_address>[^\s]+) .* \[(?P<timestamp>[^\]]+)\] (.+)$'
 7    elasticsearch.collectord.io/logs-timestampfield: timestamp
 8    elasticsearch.collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
 9spec:
10  containers:
11  - name: nginx
12    image: nginx

That results in messages

text

1ip_address | _time               | _raw
2-----------|---------------------|-------------------------------------------------
3172.17.0.1 | 2018-08-31 21:11:26 | "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
4172.17.0.1 | 2018-08-31 21:11:32 | "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
5172.17.0.1 | 2018-08-31 21:11:35 | "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Defining Event pattern

With the annotation elasticsearch.collectord.io/logs-eventpattern you can define how collectord should identify new events in the pipe. The default event pattern is defined by the collectord configuration as ^[^\s] (anything that does not start from a space character).

The default pattern works in most of the cases, but doesn’t work in some, like Java exceptions, where the call stack of the error starts on the next line, and it doesn’t start with the space character.

In example below we intentionally made a mistake in a configuration for the ElasticSearch (s-node should be a single-node) to get the error message

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: elasticsearch-pod
 5spec:
 6  containers:
 7  - name: elasticsearch
 8    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
 9    env:
10    - name: discovery.type
11      value: s-node

Results in

text

 1[2018-08-31T22:44:56,433][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/92] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
 2[2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
 3org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
 4	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]
 5	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
 6	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
 7	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
 8	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
 9	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]
10	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
11Caused by: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
12	at org.elasticsearch.discovery.DiscoveryModule.<init>(DiscoveryModule.java:129) ~[elasticsearch-6.4.0.jar:6.4.0]
13	at org.elasticsearch.node.Node.<init>(Node.java:477) ~[elasticsearch-6.4.0.jar:6.4.0]
14	at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
15	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
16	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
17	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
18	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
19	... 6 more
20[2018-08-31T22:44:56,892][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

And with the default pattern we will not have the warning line [2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] with the whole callstack.

We can define that every log event in this container should start with the [ character with the regular expression as

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: elasticsearch-pod
 5  annotations:
 6    elasticsearch.collectord.io/logs-eventpattern: '^\['
 7spec:
 8  containers:
 9  - name: elasticsearch
10    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
11    env:
12    - name: discovery.type
13      value: s-node

Note: by default, collectord joins multi-line log lines that are written in the duration of 100ms, waits maximum of 1s for the next line, and combines event in the total of 100Kb. If you see that not all multi-line log lines are joined into one event as you expect, you might need to change the configuration for the collectord under [pipe.join].

Application Logs

Sometimes it is hard or just not practical to redirect all logs from the container to stdout and stderr of the container. In these cases, you keep the logs in the container. We call them application logs. With collectord you can easily pick up these logs and forward them to ElasticSearch. No additional sidecars or processes are required inside your container.

Let’s take a look at the example below. We have a postgresql container, that redirects most of the logs to the path inside the container /var/log/postgresql. We define for this container a volume (emptyDir driver) with the name psql_logs and mount it to /var/log/postgresql/. With the annotation elasticsearch.collectord.io/volume.1-logs-name=psql_logs we tell collectord to pick up all the logs with the default glob pattern *.log* (a default glob pattern is set in the collectord configuration, and you can override it with annotation elasticsearch.collectord.io/volume.{N}-logs-glob) in the volume and forward them automatically to ElasticSearch.

When you need to forward logs from multiple volumes of the same container, you can group the settings with the same number, for example elasticsearch.collectord.io/volume.1-logs-name=psql_logs and elasticsearch.collectord.io/volume.2-logs-name=psql_logs

Example 1. Forwarding application logs

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: postgres-pod
 5  annotations:
 6    elasticsearch.collectord.io/volume.1-logs-name: 'logs'
 7spec:
 8  containers:
 9  - name: postgres
10    image: postgres
11    command:
12      - docker-entrypoint.sh
13    args:
14      - postgres
15      - -c
16      - logging_collector=on
17      - -c
18      - log_min_duration_statement=0
19      - -c
20      - log_directory=/var/log/postgresql
21      - -c
22      - log_min_messages=INFO
23      - -c
24      - log_rotation_age=1d
25      - -c
26      - log_rotation_size=10MB
27    volumeMounts:
28      - name: data
29        mountPath: /var/lib/postgresql/data
30      - name: logs
31        mountPath: /var/log/postgresql/
32  volumes:
33  - name: data
34    emptyDir: {}
35  - name: logs
36    emptyDir: {}

Example 2. Forwarding application logs with fields extraction and time parsing

With the annotations for application logs, you can define fields extraction, replace patterns, override the datastreams.

As an example, with the extraction pattern and timestamp parsing you can do

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: postgres-pod
 5  annotations:
 6    elasticsearch.collectord.io/volume.1-logs-name: 'logs'
 7    elasticsearch.collectord.io/volume.1-logs-extraction: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} [^\s]+) (.+)$'
 8    elasticsearch.collectord.io/volume.1-logs-timestampfield: 'timestamp'
 9    elasticsearch.collectord.io/volume.1-logs-timestampformat: '2006-01-02 15:04:05.000 MST'
10spec:
11  containers:
12  - name: postgres
13    image: postgres
14    command:
15      - docker-entrypoint.sh
16    args:
17      - postgres
18      - -c
19      - logging_collector=on
20      - -c
21      - log_min_duration_statement=0
22      - -c
23      - log_directory=/var/log/postgresql
24      - -c
25      - log_min_messages=INFO
26      - -c
27      - log_rotation_age=1d
28      - -c
29      - log_rotation_size=10MB
30    volumeMounts:
31      - name: data
32        mountPath: /var/lib/postgresql/data
33      - name: logs
34        mountPath: /var/log/postgresql/
35  volumes:
36  - name: data
37    emptyDir: {}
38  - name: logs
39    emptyDir: {}

That way you will extract the timestamps and remove them from the message

text

 1_time               | _raw
 22018-08-31 23:31:02 | [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
 3                    | 	  c.relname as "Name",
 4                    | 	  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
 5                    | 	  pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
 6                    | 	FROM pg_catalog.pg_class c
 7                    | 	     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
 8                    | 	WHERE c.relkind IN ('r','p','')
 9                    | 	      AND n.nspname <> 'pg_catalog'
10                    | 	      AND n.nspname <> 'information_schema'
11                    | 	      AND n.nspname !~ '^pg_toast'
12                    | 	  AND pg_catalog.pg_table_is_visible(c.oid)
13                    | 	ORDER BY 1,2;
142018-08-31 23:30:53 |  UTC [124] FATAL:  role "postgresql" does not exist

Placeholder templates in a glob pattern

In the case if you’re mounting the same volume to multiple Pods, and you want to differentiate the logs, you can now specify the placeholders in the glob configuration. For example, if you have a volume mounted to the Pod with the name my-pod and to the Pod with the name my-pod-2 you can specify the glob configuration like this {{kubernetes_pod_name}}.log, so the Collectord will be able to identify that files my-pod.log and my-pod-2.log are coming from different Pods.

On Volume Database for acknowledgements

Collectord has a database that stores the information about the files that were already processed. The database is stored by default on the host, where Collectord is running. In case, if one of the volumes is used on one host and then is
mounted to another host, the Collectord will start to process the files from the beginning. To avoid this, you can specify annotation elasticsearch.collectord.io/volume.{N}-logs-onvolumedatabase=true to enable the on volume database. In this case, Collectord creates a database in the volume root .collectord.db, so the data about processed files will be stored in the volume and will be available on any host.

Important details about this feature, you need to mount the /rootfs directory in the Collectord container with write access. By default, the /rootfs directory is mounted as read-only.

Volume types

Collectord supports two volume types for application logs: emptyDir, hostPath and persistentVolumeClaim. Collectord configuration has two settings that helps collectord to autodiscover application logs. First is the [general.kubernetes]/volumesRootDir for discovering volumes created with emptyDir, second is [input.app_logs]/root for discovering host mounts, considering that they will be mounted with a different path to collectord.

Change output destination

By default, collectord forwards all the data to ElasticSearch. You can configure containers to redirect data to devnull instead with annotation elasticsearch.collectord.io/logs-output=devnull.

By changing the default output for specific data, you can change how you forward data to ElasticSearch. Instead of forwarding all the logs by default, you can change configuration for collectord with --env "COLLECTOR__LOGS_OUTPUT=input.files__output=devnull" to specify not forward container logs by default. And define with the containers which logs you want to see in ElasticSearch with elasticsearch.collectord.io/logs-output=elasticsearch.

For example:

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  labels:
 6    app: MyApp
 7  annotations:
 8    elasticsearch.collectord.io/logs-output: 'elasticsearch'
 9spec:
10  containers:
11  - name: nginx
12    image: nginx

Additionally, if you configure multiple ElasticSearch outputs with the configuration, you can forward the data to a specific ElasticSearch Cluster as

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  labels:
 6    app: MyApp
 7  annotations:
 8    elasticsearch.collectord.io/output: 'elasticsearch::prod1'
 9spec:
10  containers:
11  - name: nginx
12    image: nginx

Forwarding logs to multiple ElasticSearch clusters simultaneously

With annotation elasticsearch.collectord.io/output you can configure multiple ElasticSearch HEC endpoints, for example elasticsearch::apps and elasticsearch::security using the comma-separated list: elasticsearch.collectord.io/output=elasticsearch::apps,elasticsearch::security. Assuming you have them defined in the ConfigMap like [output.elasticsearch::apps] and [output.elasticsearch::security].

Additionally, you can configure datastreams for the endpoints in square brackets, for example elasticsearch.collectord.io/output=elasticsearch::apps[logs-team],elasticsearch::security[logs-security].

In that case, each event will be sent to both ElasticSearch Clusters.

For example, if you want to forward logs from a specific container to multiple ElasticSearch clusters, you can use the following

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  labels:
 6    app: MyApp
 7  annotations:
 8    elasticsearch.collectord.io/logs-output: 'elasticsearch::apps[logs-team],elasticsearch::security[logs-security]'
 9spec:
10  containers:
11  - name: nginx
12    image: nginx

Logs sampling

Example 1. Random based sampling

When the application produces a high number of logs, in some cases, it could be enough to just on the sampled amount of the logs to understand how many failed requests the application has, or how it behaves. You can add an annotation for the logs to specify the percent number of the logs that should be forwarded to ElasticSearch.

In the following example, this application produces 300,000 log lines. Only about 60,000 log lines are going to be forwarded to ElasticSearch.

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: logtest
 5  annotations:
 6    elasticsearch.collectord.io/logs-sampling-percent: '20'
 7spec:
 8  restartPolicy: Never
 9  containers:
10  - name: logtest
11    image: docker.io/mffiedler/ocp-logtest:latest
12    args: [python, ocp_logtest.py,
13           --line-length=1024, --num-lines=300000, --rate=60000, --fixed-line]

Example 2. Hash-based sampling

In the situations where you want to look at the pattern for a specific user, you can specify that you want to sample logs based on the hash value, to be sure if the same key presents in two different log lines, both of them will be forwarded to ElasticSearch.

In the following example we define a key (should be a named submatch pattern) as an IP address.

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-sampling
 5  annotations:
 6    elasticsearch.collectord.io/logs-sampling-percent: '20'
 7    elasticsearch.collectord.io/logs-sampling-key: '^(?P<key>(\d+\.){3}\d+)'
 8spec:
 9  containers:
10  - name: nginx-sampling
11    image: nginx

Thruput

You can configure the thruput specifically for the container logs as

bash

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-sampling
 5  annotations:
 6    elasticsearch.collectord.io/logs-ThruputPerSecond: 128Kb
 7spec:
 8  containers:
 9  - name: nginx-sampling
10    image: nginx

In case if this container produces more than 128kb per second collectord will throttle logs.

Time correction

If you’re pre-loading a lot of logs, you might want to configure the events, that you want to skip, as they’re too old or too new

bash

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-sampling
 5  annotations:
 6    elasticsearch.collectord.io/logs-TooOldEvents: 168h
 7    elasticsearch.collectord.io/logs-TooOldEvents: 1h
 8spec:
 9  containers:
10  - name: nginx-sampling
11    image: nginx

Handling multiple containers

Pod can have multiple containers. You can define annotations for a specific container with the name prefixed the annotation. The format of the annotations is elasticsearch.collectord.io/{container_name}--{annotation}: {annotation-value}. As an example.

yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx-pod
 5  annotations:
 6    elasticsearch.collectord.io/web--logs-index: 'web'
 7    elasticsearch.collectord.io/web--logs-replace.2-search: '(?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}'
 8    elasticsearch.collectord.io/web--logs-replace.2-val: '${IPv4p1}.X.X.X'
 9    elasticsearch.collectord.io/user--logs-disabled: 'true'
10spec:
11  containers:
12  - name: web
13    image: nginx
14  - name: user
15    image: busybox
16    args: [/bin/sh, -c,
17           'while true; do wget -qO- localhost:80 &> /dev/null; sleep 5; done']

Cluster level annotations

You can apply annotations to Pods on the cluster level with Configuration in api group collectord.io/v1. For example

yaml

 1apiVersion: "collectord.io/v1"
 2kind: Configuration
 3metadata:
 4  name: apply-to-all-nginx
 5  annotations:
 6    elasticsearch.collectord.io/nginx--logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
 7    elasticsearch.collectord.io/nginx--logs-replace.1-val: ''
 8    elasticsearch.collectord.io/nginx--logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
 9    elasticsearch.collectord.io/nginx--logs-hashing.1-function: 'fnv-1a-64'
10spec:
11  kubernetes.container.image: "^nginx(:.*)?$"

This configuration will be applied to all containers that use the image with nginx in the name (examples are nginx:latest or nginx:1.0).

In the spec of the Configuration you include selectors based on the meta fields that we forward to ElasticSearch, which can include fields like container.image.name, kubernetes.container.name, kubernetes.daemonset.name, kubernetes.namespace, kubernetes.pod.name, etc. When you specify multiple fields in the spec, all regexes should match.

Forcing Cluster Level Annotations

If you already have an annotation, for example, elasticsearch.collectord.io/index=foo defined on Namespace, Deployment, or Pod, and if you’re trying to apply this annotation from Cluster Level Configuration as elasticsearch.collectord.io/index=bar, the one from the objects will take priority.

With the force modifier you can override those annotations, even if you have them defined on the objects.

yaml

1apiVersion: "collectord.io/v1"
2kind: Configuration
3metadata:
4  name: apply-to-all-nginx
5  annotations:
6    elasticsearch.collectord.io/index=bar
7spec:
8  kubernetes_container_image: "^nginx(:.*)?$"
9force: true

NOTE: if you have an annotation defined in the namespace as elasticsearch.collectord.io/logs-index=foo, it will still take priority over index=bar, as logs-index=foo is type specific.

Troubleshooting

Check the collectord logs for warning messages about the annotations, you can find if you made a misprint in the annotations if you see warnings like

text

1WARN 2018/08/31 21:05:33.122978 core/input/annotations.go:76: invalid annotation ...

Some pipes, like fields extraction and time-parsing pipes adds an error in the field collectord_errors, so you can identify when some events failed to be processed by this pipe.

Describe command

Please use Collectord describe command to see how annotations are applied to a specific pod or container.

See Troubleshooting -> Describe.

Reference

For the full list of every annotation grouped by datatype, see Annotations reference.

Annotations

Overriding datastreams

Overriding datastream for specific events

Replace patterns in events

Example 1. Replacing IPv4 addresses with X.X.X.X

Example 2. Dropping messages

Example 3. Whitelisting the messages

Hashing values in logs

Escaping terminal sequences, including terminal colors

Extracting fields from the container logs

Nested Objects

Datastreams and index templates

Example 1. Extracting the timestamp

Example 2. Extracting the fields

Defining Event pattern

Application Logs

Example 1. Forwarding application logs

Example 2. Forwarding application logs with fields extraction and time parsing

Placeholder templates in a glob pattern

On Volume Database for acknowledgements

Volume types

Change output destination

Forwarding logs to multiple ElasticSearch clusters simultaneously

Logs sampling

Example 1. Random based sampling

Example 2. Hash-based sampling

Thruput

Time correction

Handling multiple containers

Cluster level annotations

Forcing Cluster Level Annotations

Troubleshooting

Describe command

Reference

Example 1. Replacing IPv4 addresses with `X.X.X.X`