Container Annotations

You can define annotations for the containers, to change how collector forwards data to Splunk. Annotations can be used to tell collector where to find the application logs.

The list of all the available annotations for the containers is below.

Collectord accepts annotations with format collectord.io/{annotation}={value} and io.collectord.{annotations}={value}.

Starting from version 5.9 you can define a subdomain for annotations under [general]annotationsSubdomain, that allows you to deploy multiple collectord instances and configure them with different annotations. If you want to use an annotation that applies to all collectord instance you can use a subdomain collectord.collectord.io/{annotation}={value}

Overriding indexes

Using annotations you can override to which index container should redirect the logs and metrics. You can define one index for the whole container with collectord.io/index or specific indices for container logs collectord.io/logs-index, container stats collectord.io/stats-index and process stats collectord.io/procstats-index.

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-index=project1_logs' \
       --label 'collectord.io/stats-index=project1_stats' \
       --label 'collectord.io/procstats-index=project1_stats' \
       --label 'collectord.io/netstats-index=project1_stats' \
       --label 'collectord.io/nettable-index=project1_stats' \
       nginx

Similarly you can override source, type and host.

Overriding index, source and type for specific events

Available since version 5.2

In the case when your container is running multiple processes, sometimes you want to override source or type just for specific events in the container (or application) logs. You can do that with the override annotations.

For example we will use the nginx image with logs

172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
2018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

If we want to override source of the web logs and keep all other logs with the predefined source

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-override.1-match=^(\d{1,3}\.){3}\d{1,3}' \
       --label 'collectord.io/logs-override.1-source=/docker/nginx/web-log' \
       nginx

The collector will override source for matched events, so with our example you will end up with events similar to

source                 | event
------------------------------------------------------------------------------------------------------------------------
/docker/nginx/web-log  | 172.17.0.1 - - [12/Oct/2018:22:38:05 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
/docker/550...stderr   | 2018/10/12 22:38:15 [error] 8#8: *2 open() "/usr/share/nginx/html/a.txt" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /a.txt HTTP/1.1", host: "localhost:32768"
/docker/nginx/web-log  | 172.17.0.1 - - [12/Oct/2018:22:38:15 +0000] "GET /a.txt HTTP/1.1" 404 153 "-" "curl/7.54.0" "-"

Replace patterns in events

You can define replace patterns with the annotations. That allows you to hide sensitive information, or drop unimportant information from the messages.

Replace patterns for container logs are configured with pair of annotations grouped with the same number collectord.io/logs-replace.1-search and collectord.io/logs-replace.2-val, first specifies the search pattern as a regular expression, second a replace pattern. In replace patterns you can use placeholders for matches, like $1 or $name for named patterns.

We are using Go regular expression library for replace pipes. You can find more information about the syntax at Package regexp and re2 syntax. We recommend to use https://regex101.com for testing your patterns (set the Flavor to golang).

Using nginx as an example, our logs have a default pattern like

172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 1. Replacing IPv4 addresses with `X.X.X.X`

If we want to hide an IP address from the logs by replacing all IPv4 addresses with X.X.X.X

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-replace.1-search=(\d{1,3}\.){3}\d{1,3}' \
       --label 'collectord.io/logs-replace.1-val=X.X.X.X' \
       nginx

The result of this replace pattern will be in Splunk

X.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

You can also keep the first part of the IPv4 with

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-replace.1-search=(?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}' \
       --label 'collectord.io/logs-replace.1-val=${IPv4p1}.X.X.X' \
       nginx

That results in

172.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 2. Dropping messages

With the replace patterns you can drop messages that you don't want to see in Splunk. With the example below we drop all log messages resulted from GET requests with 200 response

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-replace.1-search=^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$' \
       --label 'collectord.io/logs-replace.1-val=' \
       --label 'collectord.io/logs-replace.2-search=(\d{1,3}\.){3}\d{1,3}' \
       --label 'collectord.io/logs-replace.2-val=X.X.X.X' \
       nginx

In this example we have two replace pipes. The apply in the alphabetical order (replace.1 comes first, before the replace.2).

X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 3. Whitelisting the messages

With the whitelist annotation you can configure pattern for the log messages, and only messages that match this pattern will be forwarded to Splunk

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-whitelist=((DELETE)|(POST))' \
       nginx

Hashing values in logs

Available since version 5.3

To hide sensitive data, you can use replace patterns or hashing functions.

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-hashing.1-match=(\d{1,3}\.){3}\d{1,3}' \
       --label 'collectord.io/logs-hashing.1-function=fnv-1a-64' \
       nginx

This example will replace values that look like an IP address in the string

172.17.0.1 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

With the hashed value, in our example using the algorithm fnv-1a-64

gqsxydjtZL4 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

Collectord supports a variety of hashing functions including cryptographic hashing functions. The list of supported functions and their performance is listed below (performance in the nanoseconds per operation to hash two IP addresses in a string source: 127.0.0.1, destination: 10.10.1.99)

| Function          | ns / op |
-------------------------------
| adler-32          |    1713 |
| crc-32-ieee       |    1807 |
| crc-32-castagnoli |    1758 |
| crc-32-koopman    |    1753 |
| crc-64-iso        |    1739 |
| crc-64-ecma       |    1740 |
| fnv-1-64          |    1711 |
| fnv-1a-64         |    1711 |
| fnv-1-32          |    1744 |
| fnv-1a-32         |    1738 |
| fnv-1-128         |    1852 |
| fnv-1a-128        |    1836 |
| md5               |    2032 |
| sha1              |    2037 |
| sha256            |    2220 |
| sha384            |    2432 |
| sha512            |    2516 |

Escaping terminal sequences, including terminal colors

Some containers does not turn off terminal colors automatically, when they run inside docker. For example if you run container with attached tty and define that you want to see colors

docker run -it ubuntu ls --color=auto /
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

You can find messages similar to below in Splunk

[01;34mboot[0m  [01;34metc[0m  [01;34mlib[0m   [01;34mmedia[0m  [01;34mopt[0m  [01;34mroot[0m  [01;34msbin[0m  [01;34msys[0m  [01;34musr[0m
[0m[01;34mbin[0m   [01;34mdev[0m  [01;34mhome[0m  [01;34mlib64[0m  [01;34mmnt[0m  [01;34mproc[0m  [01;34mrun[0m   [01;34msrv[0m  [30;42mtmp[0m  [01;34mvar[0m

You can easily escape them with the annotation collectord.io/logs-escapeterminalsequences=true

docker run -it \
    --label 'collectord.io/logs-escapeterminalsequences=true' \
    ubuntu ls --color=auto /

That way you will see logs in Splunk as you would expect

bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

In the collector configuration file you can find [input.files]/stripTerminalEscapeSequencesRegex and [input.files]/stripTerminalEscapeSequences that defines default regexp used for removing terminal escape sequences and default value if collector should strip terminal escape sequences (defaults to false).

Extracting fields from the container logs

You can use fields extraction, that will allow you to extract timestamps from the messages, extract fields that will be indexed with Splunk to speed up the search.

Using the same example with nginx we can define fields extraction for some of the fields.

172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Important note, that first unnamed pattern is used as the message for the event. If you want to override it, you can use the annotations (available in 5.18+) collectord.io/logs-extractionMessageField to use a specific pattern as a message field.

Example 1. Extracting the timestamp

Assuming we want to keep whole message as is, and extract just a timestamp. We can define the extraction pattern with the regexp. Specify that the timestampfield is timestamp and define the timestampformat.

We use Go time parsing library, that defines the format with the specific date Mon Jan 2 15:04:05 MST 2006. See Go documentation for details.

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-extraction=^(.*\[(?P<timestamp>[^\]]+)\].+)$' \
       --label 'collectord.io/logs-timestampfield=timestamp' \
       --label 'collectord.io/logs-timestampformat=02/Jan/2006:15:04:05 -0700' \
       nginx

In that way you will get messages in Splunk with the exact timestamp as specified in your container logs.

Example 2. Extracting the fields

If you want to extract some fields, and keep the message shorter, as an example, if you have extracted the timestamps, there is no need for you to keep the timestamp in the raw message. In the example below we extract the ip_address address as a field, timestamp and keep the rest as a raw message.

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-extraction=^(?P<ip_address>[^\s]+) .* \[(?P<timestamp>[^\]]+)\] (.+)$' \
       --label 'collectord.io/logs-timestampfield=timestamp' \
       --label 'collectord.io/logs-timestampformat=02/Jan/2006:15:04:05 -0700' \
       nginx

That results in messages

ip_address | _time               | _raw
-----------|---------------------|-------------------------------------------------
172.17.0.1 | 2018-08-31 21:11:26 | "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 | 2018-08-31 21:11:32 | "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 | 2018-08-31 21:11:35 | "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Defining Event pattern

With the annotation collectord.io/logs-eventpattern you can define how collector should identify new events in the pipe. The default event pattern is defined by the collector configuration as ^[^\s] (anything that does not start from a space character).

The default pattern works in most of the cases, but does not work in some, like Java exceptions, where the call stack of the error starts on the next line, and it does not start with the space character.

We intentionally made a mistake in a configuration for the ElasticSearch (s-node should be a single-node) to get the error message

docker run --env "discovery.type=s-node" docker.elastic.co/elasticsearch/elasticsearch:6.4.0

Results in

[2018-08-31T22:44:56,433][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/92] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
    at org.elasticsearch.discovery.DiscoveryModule.<init>(DiscoveryModule.java:129) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.node.Node.<init>(Node.java:477) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
    ... 6 more
[2018-08-31T22:44:56,892][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

And with the default pattern we will not have the warning line [2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] with the whole callstack.

In example below we can define that every log event in this container should start with the [ character with the regular expression as

docker run --env "discovery.type=s-node" \
    --label 'collectord.io/logs-eventpattern=^\[' \
    docker.elastic.co/elasticsearch/elasticsearch:6.4.0

Note: by default collectord joins multi-line log lines that written in the duration of 100ms, waits maximum of 1s for the next line, and combines event in the total of 100Kb. If you see that not all multi-line log lines are joined into one event as you expect, you might need to change the configuration for the collectord under [pipe.join].

Application Logs

Sometimes it is hard or just not practical to redirect all logs from the container to stdout and stderr of the container. In that cases you keep the logs in the container. We call them application logs. With collector you can easily pick up these logs and forward them to Splunk. No additional sidecars or processes required inside your container.

Let's take a look on the example below. We have a postgresql container, that redirects most of the logs to the path inside the container /var/log/postgresql. We define for this container a volume (local driver) with the name psql_logs and mount it to /var/log/postgresql/. With the annotation collectord.io/volume.1-logs-name=psql_logs we tell collector to pick up all the logs with the default glob pattern *.log* (default glob pattern is set int the collector configuration, and you can override it with annotation collectord.io/volume.{N}-logs-glob) in the volume and forward them automatically to Splunk.

When you need to forward logs from multiple volumes of the same container you can group the settings with the same number, for example collectord.io/volume.1-logs-name=psql_logs and collectord.io/volume.2-logs-name=psql_logs

Example 1. Forwarding application logs

docker run -d \
    --volume psql_data:/var/lib/postgresql/data \
    --volume psql_logs:/var/log/postgresql/ \
    --label 'collectord.io/volume.1-logs-name=psql_logs' \
    postgres:10.4 \
    docker-entrypoint.sh postgres -c logging_collector=on -c log_min_duration_statement=0 -c log_directory=/var/log/postgresql -c log_min_messages=INFO -c log_rotation_age=1d -c log_rotation_size=10MB

In the example above the logs from the container will have a source, similar to psql_logs:postgresql-2018-08-31_232946.log.

2018-08-31 23:31:02.034 UTC [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
      c.relname as "Name",
      CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
      pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
    FROM pg_catalog.pg_class c
         LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
    WHERE c.relkind IN ('r','p','')
          AND n.nspname <> 'pg_catalog'
          AND n.nspname <> 'information_schema'
          AND n.nspname !~ '^pg_toast'
      AND pg_catalog.pg_table_is_visible(c.oid)
    ORDER BY 1,2;
2018-08-31 23:30:53.490 UTC [124] FATAL:  role "postgresql" does not exist

Example 2. Forwarding application logs with fields extraction and time parsing

With the annotations for application logs you can define fields extraction, replace patterns, override the indexes, sources and hosts.

As an example, with the extraction pattern and timestamp parsing you can do

docker run -d \
    --volume psql_data:/var/lib/postgresql/data \
    --volume psql_logs:/var/log/postgresql/ \
    --label 'collectord.io/volume.1-logs-name=psql_logs' \
    --label 'collectord.io/volume.1-logs-extraction=^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} [^\s]+) (.+)$' \
    --label 'collectord.io/volume.1-logs-timestampfield=timestamp' \
    --label 'collectord.io/volume.1-logs-timestampformat=2006-01-02 15:04:05.000 MST' \
    postgres:10.4 \
    docker-entrypoint.sh postgres -c logging_collector=on -c log_min_duration_statement=0 -c log_directory=/var/log/postgresql -c log_min_messages=INFO -c log_rotation_age=1d -c log_rotation_size=10MB

That way you will extract the timestamps and remove them from the _raw message

_time               | _raw
2018-08-31 23:31:02 | [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
                    |     c.relname as "Name",
                    |     CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
                    |     pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
                    |   FROM pg_catalog.pg_class c
                    |        LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    |   WHERE c.relkind IN ('r','p','')
                    |         AND n.nspname <> 'pg_catalog'
                    |         AND n.nspname <> 'information_schema'
                    |         AND n.nspname !~ '^pg_toast'
                    |     AND pg_catalog.pg_table_is_visible(c.oid)
                    |   ORDER BY 1,2;
2018-08-31 23:30:53 |  UTC [124] FATAL:  role "postgresql" does not exist

Volume types

Collector supports two volume types for application logs: local and host mount. Collector configuration has two settings that helps collector to autodiscover application logs. First is the [general.docker]/dockerRootFolder for discovering volumes created with local driver, second is [input.app_logs]/root for discovering host mounts, considering that they will be mounted with different path to collector.

Change output destination

By default collector forwards all the data to Splunk. You can configure containers to redirect data to devnull instead with annotation collectord.io/output=devnull. This annotations changes output for all the data related to this container. Alternatively you can change output only for logs with the annotation collectord.io/logs-output=devnull.

By changing default output for specific data you can change how you forward data to Splunk. Instead of forwarding all the logs by default, you can change configuration for collector with --env "COLLECTOR__LOGS_OUTPUT=input.files__output=devnull" to specify not forward container logs by default. And define with the containers which logs you want to see in Splunk with collectord.io/logs-output=splunk.

For example:

docker run --rm \
       --publish 80 \
       --label 'collectord.io/logs-output=splunk' \
       nginx

Additionally, if you configured multiple Splunk outputs with the configuration (see Support for multiple Splunk clusters) you can forward the data to a specific Splunk Cluster as

docker run --rm \
       --publish 80 \
       --label 'collectord.io/output=splunk::prod1' \
       nginx

Logs sampling

Available since collectorfordocker v5.6

Example 1. Random based sampling

When the application produces a high amount of logs, in some cases it could be enough to just on the sampled amount of the logs to understand how many failed requests the application has, or how it behaves. You can add an annotation for the logs to specify the percent amount of the logs that should be forwarded to splunk.

In the following example, this application produces 300,000 log lines. Only about 60,000 log lines are going to be forwarded to Splunk.

docker run -d --rm \
    --label 'collectord.io/logs-sampling-percent=20' \
    docker.io/mffiedler/ocp-logtest:latest \
    python ocp_logtest.py --line-length=1024 --num-lines=300000 --rate 60000 --fixed-line

Example 2. Hash-based sampling

In the situations where you want to look at the pattern for a specific user, you can specify that you want to sample logs based on the hash value, to be sure if the same key presents in two different log lines, both of them will be forwarded to Splunk.

In the following example we define a key (should be a named submatch pattern) as an IP address.

docker run -d --rm \
    --label 'collectord.io/logs-sampling-percent=20' \
    --label 'collectord.io/logs-sampling-key=^(?P<key>(\d+\.){3}\d+)' \
    nginx

Thruput

Available since collectorfordocker v5.10.252

You can configure the thruput specifically for the container logs as

docker run -d --rm \
    --label 'collectord.io/logs-ThruputPerSecond=128Kb' \
    nginx

In case if this container produces more than 128kb per second collectord will throttle logs.

Time correction

Available since collectorfordocker v5.10.252

If you are pre-loading a lot of logs, you might want to configure the events, that you want to skip, as they are too old or too new

docker run -d --rm \
    --label 'collectord.io/logs-TooOldEvents=168h' \
    --label 'collectord.io/logs-TooNewEvents=1h' \
    nginx

Troubleshooting

Check the collector logs for warning messages about the annotations, you can find if you made a misprint in the annotations if you see warnings like

WARN 2018/08/31 21:05:33.122978 core/input/annotations.go:76: invalid annotation ...

Some pipes, like fields extraction and time parsing pipes adds a error in the field collectord_errors, so you can identify when some events failed to be processed by this pipe.

Reference

General annotations
- collectord.io/index - change the index for all the data forwarded for this container (metrics, container logs, application logs)
- collectord.io/source - change the source for all the data forwarded for this container (metrics, container logs, application logs)
- collectord.io/type - change the sourcetype for all the data forwarded for this container (metrics, container logs, application logs)
- collectord.io/host - change the host for all the data forwarded for this container (metrics, container logs, application logs)
- collectord.io/output - (5.2+) change the output to devnull or splunk
- collectord.io/userfields.{fieldname} - (5.15.300+) attach custom fields to events
Annotations for container logs
- collectord.io/logs-index - change the index for the container logs forwarded from this container
- collectord.io/logs-source - change the source for the container logs forwarded from this container
- collectord.io/logs-type - change the sourcetype for the container logs forwarded from this container
- collectord.io/logs-host - change the host for the container logs forwarded from this container
- collectord.io/logs-eventpattern - set the regex identifying the event start pattern for container logs
- collectord.io/logs-replace.{N}-search - define the search pattern for the replace pipe
- collectord.io/logs-replace.{N}-val - define the replace pattern for the replace pipe
- collectord.io/logs-hashing.{N}-match - (5.3+) the regexp for a matched value
- collectord.io/logs-hashing.{N}-function - (5.3+) hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
- collectord.io/logs-extraction - define the regexp for fields extraction
- collectord.io/logs-extractionMessageField - (5.18+) specify the field name for the message (by default first unnamed ground in regexp)
- collectord.io/logs-timestampfield - define the field for timestamp (after fields extraction)
- collectord.io/logs-timestampformat - define the timestamp format
- collectord.io/logs-timestampsetmonth - define if month should be set to current for timestamp
- collectord.io/logs-timestampsetday - define if day should be set to current for timestamp
- collectord.io/logs-timestamplocation - define timestamp location if not set by format
- collectord.io/logs-joinpartial - join partial events
- collectord.io/logs-joinmultiline - (5.3+) join multiline logs (default value depends on [pipe.join] disabled)
- collectord.io/logs-escapeterminalsequences - escape terminal sequences (including colors)
- collectord.io/logs-override.{N}-match - (5.2+) match for override pattern
- collectord.io/logs-override.{N}-index - (5.2+) override index for matched events
- collectord.io/logs-override.{N}-source - (5.2+) override source for matched events
- collectord.io/logs-override.{N}-type - (5.2+) override type for matched events
- collectord.io/logs-output - (5.2+) change the output to devnull or splunk (this annotation cannot be specified for stderr and stdout)
- collectord.io/logs-disabled - (5.3+) disable any log processing for this container (this annotation cannot be specified for stderr and stdout)
- collectord.io/logs-sampling-percent - (5.6+) specify the % value of logs that should be forwarded to splunk
- collectord.io/logs-sampling-key - (5.6+) regexp pattern to specify the key for the sampling based on hash values
- collectord.io/logs-ThruputPerSecond - (5.10.252+) set the thruput for this container, maximum amount of logs per second, for example 128Kb, 1024b
- collectord.io/logs-TooOldEvents - (5.10.252+) duration of events from now to past that are considered too old and should be ignored, for example 168h, 24h
- collectord.io/logs-TooNewEvents - (5.10.252+) duration of events from now to the future that are considered too new and should be ignored, for example 1h, 30m
- collectord.io/logs-whitelist - (5.14.284+) allow to configure pattern for log messages, only log messages matching this pattern will be forwarded to Splunk
- collectord.io/logs-userfields.{fieldname} - (5.15.300+) attach custom fields to events
- Specific for stdout, with the annotations below you can define configuration specific for stdout
  - collectord.io/stdout-logs-index
  - collectord.io/stdout-logs-source
  - collectord.io/stdout-logs-type
  - collectord.io/stdout-logs-host
  - collectord.io/stdout-logs-eventpattern
  - collectord.io/stdout-logs-replace.{N}-search
  - collectord.io/stdout-logs-replace.{N}-val
  - collectord.io/stdout-logs-hashing.{N}-match - (5.3+)
  - collectord.io/stdout-logs-hashing.{N}-function - (5.3+)
  - collectord.io/stdout-logs-extraction
  - collectord.io/stdout-logs-extractionMessageField - (5.18+)
  - collectord.io/stdout-logs-timestampfield
  - collectord.io/stdout-logs-timestampformat
  - collectord.io/stdout-logs-timestampsetmonth
  - collectord.io/stdout-logs-timestampsetday
  - collectord.io/stdout-logs-timestamplocation
  - collectord.io/stdout-logs-joinpartial
  - collectord.io/stdout-logs-joinmultiline - (5.3+)
  - collectord.io/stdout-logs-escapeterminalsequences
  - collectord.io/stdout-logs-override.{N}-match - (5.2+)
  - collectord.io/stdout-logs-override.{N}-index - (5.2+)
  - collectord.io/stdout-logs-override.{N}-source - (5.2+)
  - collectord.io/stdout-logs-override.{N}-type - (5.2+)
  - collectord.io/stdout-logs-sampling-percent - (5.6+)
  - collectord.io/stdout-logs-sampling-key - (5.6+)
  - collectord.io/stdout-logs-ThruputPerSecond - (5.10.252+)
  - collectord.io/stdout-logs-TooOldEvents - (5.10.252+)
  - collectord.io/stdout-logs-TooNewEvents - (5.10.252+)
  - collectord.io/stdout-logs-whitelist - (5.14.284+)
- Specific for stderr, with the annotations below you can define configuration specific for stderr
  - collectord.io/stderr-logs-index
  - collectord.io/stderr-logs-source
  - collectord.io/stderr-logs-type
  - collectord.io/stderr-logs-host
  - collectord.io/stderr-logs-eventpattern
  - collectord.io/stderr-logs-replace.{N}-search
  - collectord.io/stderr-logs-replace.{N}-val
  - collectord.io/stderr-logs-hashing.{N}-match - (5.3+)
  - collectord.io/stderr-logs-hashing.{N}-function - (5.3+)
  - collectord.io/stderr-logs-extraction
  - collectord.io/stdout-logs-extractionMessageField - (5.18+)
  - collectord.io/stderr-logs-timestampfield
  - collectord.io/stderr-logs-timestampformat
  - collectord.io/stderr-logs-timestampsetmonth
  - collectord.io/stderr-logs-timestampsetday
  - collectord.io/stderr-logs-timestamplocation
  - collectord.io/stderr-logs-joinpartial
  - collectord.io/stderr-logs-joinmultiline - (5.3+)
  - collectord.io/stderr-logs-escapeterminalsequences
  - collectord.io/stderr-logs-override.{N}-match - (5.2+)
  - collectord.io/stderr-logs-override.{N}-index - (5.2+)
  - collectord.io/stderr-logs-override.{N}-source - (5.2+)
  - collectord.io/stderr-logs-override.{N}-type - (5.2+)
  - collectord.io/stderr-logs-sampling-percent - (5.6+)
  - collectord.io/stderr-logs-sampling-key - (5.6+)
  - collectord.io/stderr-logs-ThruputPerSecond - (5.10.252+)
  - collectord.io/stderr-logs-TooOldEvents - (5.10.252+)
  - collectord.io/stderr-logs-TooNewEvents - (5.10.252+)
  - collectord.io/stderr-logs-whitelist - (5.14.284+)
Annotations for container stats
- collectord.io/stats-index - change the index for the container metrics forwarded from this container
- collectord.io/stats-source - change the source for the container metrics forwarded from this container
- collectord.io/stats-type - change the sourcetype for the container metrics forwarded from this container
- collectord.io/stats-host - change the host for the container metrics forwarded from this container
- collectord.io/stats-output - (5.2+) change the output to devnull or splunk
- collectord.io/stats-interval - (5.14+) override default interval how often to collect stats
- collectord.io/stats-userfields.{fieldname} - (5.15.300+) attach custom fields to events
Annotations for container processes stats
- collectord.io/procstats-index - change the index for the container process metrics forwarded from this container
- collectord.io/procstats-source - change the source for the container process metrics forwarded from this container
- collectord.io/procstats-type - change the type for the container process metrics forwarded from this container
- collectord.io/procstats-host - change the host for the container process metrics forwarded from this container
- collectord.io/procstats-output - (5.2+) change the output to devnull or splunk
- collectord.io/procstats-userfields.{fieldname} - (5.15.300+) attach custom fields to events
Annotations for container network stats
- collectord.io/netstats-index - change the index for the container network metrics forwarded from this Pod
- collectord.io/netstats-source - change the source for the container network metrics forwarded from this Pod
- collectord.io/netstats-type - change the type for the container network metrics forwarded from this Pod
- collectord.io/netstats-host - change the host for the container network metrics forwarded from this Pod
- collectord.io/netstats-output - (5.2+) change the output to devnull or splunk
- collectord.io/netstats-interval - (5.14+) override default interval how often to collect stats
- collectord.io/netstats-userfields.{fieldname} - (5.15.300+) attach custom fields to events
Annotations for container network socket table
- collectord.io/nettable-index - change the index for the container network socket table forwarded from this Pod
- collectord.io/nettable-source - change the source for the container network socket table forwarded from this Pod
- collectord.io/nettable-type - change the type for the container network socket table forwarded from this Pod
- collectord.io/nettable-host - change the host for the container network socket table forwarded from this Pod
- collectord.io/nettable-output - (5.2+) change the output to devnull or splunk
- collectord.io/nettable-interval - (5.14+) override default interval how often to collect stats
- collectord.io/nettable-userfields.{fieldname} - (5.15.300+) attach custom fields to events
Annotations for application logs
- collectord.io/volume.{N}-logs-name - name of the volume attached to container
- collectord.io/volume.{N}-logs-index - target index for logs forwarded from the volume
- collectord.io/volume.{N}-logs-source - change the source for logs forwarded from the volume
- collectord.io/volume.{N}-logs-type - change the type for logs forwarded from the volume
- collectord.io/volume.{N}-logs-host - change the host for logs forwarded from the volume
- collectord.io/volume.{N}-logs-eventpattern - change the event pattern defining new event for logs forwarded from the volume
- collectord.io/volume.{N}-logs-replace.{N}-search - specify the regex search for replace pipe for the logs
- collectord.io/volume.{N}-logs-replace.{N}-val - specify the regex replace pattern for replace pipe for the logs
- collectord.io/volume.{N}-logs-hashing.{N}-match - (5.3+) the regexp for a matched value
- collectord.io/volume.{N}-logs-hashing.{N}-function - (5.3+) hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
- collectord.io/volume.{N}-logs-extraction - specify the fields extraction with the regex the logs
- collectord.io/volume.{N}-logs-extractionMessageField - (5.18+) specify the field name for the message (by default first unnamed ground in regexp)
- collectord.io/volume.{N}-logs-timestampfield - specify the timestamp field
- collectord.io/volume.{N}-logs-timestampformat - specify the format for timestamp field
- collectord.io/volume.{N}-logs-timestampsetmonth - define if month should be set to current for timestamp
- collectord.io/volume.{N}-logs-timestampsetday - define if day should be set to current for timestamp
- collectord.io/volume.{N}-logs-timestamplocation - define timestamp location if not set by format
- collectord.io/volume.{N}-logs-glob - set the glob pattern for matching logs
- collectord.io/volume.{N}-logs-match - set the regexp pattern for matching logs
- collectord.io/volume.{N}-logs-recursive - set if walker should walk the directory recursive
- collectord.io/volume.{N}-logs-override.{N}-match - (5.2+) match for override pattern
- collectord.io/volume.{N}-logs-override.{N}-index - (5.2+) override index for matched events
- collectord.io/volume.{N}-logs-override.{N}-source - (5.2+) override source for matched events
- collectord.io/volume.{N}-logs-override.{N}-type - (5.2+) override type for matched events
- collectord.io/volume.{N}-logs-sampling-percent - (5.6+) specify the % value of logs that should be forwarded to splunk
- collectord.io/volume.{N}-logs-sampling-key - (5.6+) regexp pattern to specify the key for the sampling based on hash values
- collectord.io/volume.{N}-logs-ThruputPerSecond - (5.10.252+) set the thruput for this container, maximum amount of logs per second, for example 128Kb, 1024b
- collectord.io/volume.{N}-logs-TooOldEvents - (5.10.252+) duration of events from now to past that are considered too old and should be ignored, for example 168h, 24h
- collectord.io/volume.{N}-logs-TooNewEvents - (5.10.252+) duration of events from now to the future that are considered too new and should be ignored, for example 1h, 30m
- collectord.io/volume.{N}-logs-whitelist - (5.14.284+) allow to configure pattern for log messages, only log messages matching this pattern will be forwarded to Splunk
- collectord.io/volume.{N}-logs-userfields.{fieldname} - (5.15.300+) attach custom fields to events
- collectord.io/volume.{N}-logs-maxholdafterclose - (5.18.381+) how long Collectord can hold file descriptors open for files in the volume after container is terminated (duration 5s, 1800s)

Links

Installation
- Start monitoring your docker environments in under 10 minutes.
- Automatically forward host, container and application logs.
- Test our solution with the embedded 30 days evaluation license.
Collector Configuration
- Collector configuration reference.
- Build custom image on top collector image with embedded configuration.
Container Annotations
- Forwarding application logs.
- Multi-line container logs.
- Fields extraction for application and container logs (including timestamp extractions).
- Hiding sensitive data, stripping terminal escape codes and colors.
Configuring Splunk Indexes
- Using not default HTTP Event Collector index.
- Configure the Splunk application to use not searchable by default indexes.
Splunk fields extraction for container logs
- Configure search-time fields extractions for container logs.
- Container logs source pattern.
Configurations for Splunk HTTP Event Collector
- Configure multiple HTTP Event Collector endpoints for Load Balancing and Fail-overs.
- Secure HTTP Event Collector endpoint.
- Configure the Proxy for HTTP Event Collector endpoint.
Collecting metrics from Prometheus format
- Configure collector to forward metrics from the services in Prometheus format.
Monitoring multiple clusters
- Learn how you can monitor multiple clusters.
- Learn how to set up ACL in Splunk.
Streaming Docker Objects from API Engine
- Learn how you can poll docker containers and images and forward them to Splunk.
License Server
- Learn how you can configure remote License URL for Collectord.
Alerts
Troubleshooting
Release History
Upgrade instructions
Security
FAQ and the common questions
License agreement
Pricing
Contact

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.

Monitoring Docker - Version 5

Container Annotations

Overriding indexes

Overriding index, source and type for specific events

Replace patterns in events

Example 1. Replacing IPv4 addresses with `X.X.X.X`

Example 2. Dropping messages

Example 3. Whitelisting the messages

Hashing values in logs

Escaping terminal sequences, including terminal colors

Extracting fields from the container logs

Example 1. Extracting the timestamp

Example 2. Extracting the fields

Defining Event pattern

Application Logs

Example 1. Forwarding application logs

Example 2. Forwarding application logs with fields extraction and time parsing

Volume types

Change output destination

Logs sampling

Example 1. Random based sampling

Example 2. Hash-based sampling

Thruput

Time correction

Troubleshooting

Reference

Links

About Outcold Solutions

Monitoring Docker - Version 5

Container Annotations

Overriding indexes

Overriding index, source and type for specific events

Replace patterns in events

Example 1. Replacing IPv4 addresses with X.X.X.X

Example 2. Dropping messages

Example 3. Whitelisting the messages

Hashing values in logs

Escaping terminal sequences, including terminal colors

Extracting fields from the container logs

Example 1. Extracting the timestamp

Example 2. Extracting the fields

Defining Event pattern

Application Logs

Example 1. Forwarding application logs

Example 2. Forwarding application logs with fields extraction and time parsing

Volume types

Change output destination

Logs sampling

Example 1. Random based sampling

Example 2. Hash-based sampling

Thruput

Time correction

Troubleshooting

Reference

Links

About Outcold Solutions

Example 1. Replacing IPv4 addresses with `X.X.X.X`