Outcold Solutions LLC

Monitoring Docker - Version 3

You are looking at documentation for an older release. See the current release documentation.

Monitoring Docker Configuration

Mapped directories:

  • /data/ is a folder to keep the state of the read file positions.
  • /var/lib/docker/containers/ source of the JSON log files.
  • /var/run/docker.sock UNIX socket to load events and containers metadata.
  • /sys/fs/cgroup cgroup file system for metrics.
  • /proc proc file system for metrics.

Collector configuration

Configuration file collector.conf file is mapped to the /config/collector.conf file. Use the latest version of collector.conf, make necessary changes and embed this file in the container. You can do that by creating your image on top of our image or by mounting your configuration directly to the container.

All the values can be overridden using environment values with the format as specified below

COLLECTOR__{ANY_NAME}={section}__{key}={value}

Configurations with environment variables are the simplest way to explore and debug quickly, but we recommend to write your configuration file based on the default provided with collector.conf.

Join Rules

By default collector joins all messages with previous if they start with spaces. Below you can find how to specify a custom rule on the example of java application.

If this is a sample of the application logs.

[2017-09-04T06:28:05,664][WARN ][MyComponent]
java.security.AccessControlException: access denied
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
  at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
[2017-09-04T06:28:05,664][WARN ][MyComponent] another message

You can specify the join rules, where you configure that you want to match all containers with the name that contains my_app in their name, and pattern for the new message should match regex ^\[\d{4}-.

[pipe.join::my_app]
matchRegex.kubernetes_container_name = .+my_app.+
patternRegex = ^\[\d{4}-

Comparing with other solutions

Comparing with Splunk Logging Driver

  • Collector is based on native JSON log driver. Does not require Splunk Logging Driver. That allows you to read logs with docker logs command as usual.
  • At least once delivery. All the logs are stored in JSON files, in case if your connection to Splunk is down
    • collector keeps retrying to forward data, while these files are on disk.
  • Support for multi-line events. Specify rules for containers on how you identify new events. Java call stacks will be one event again.
  • Enriches log lines with image name, image id, container name, container id, and labels.
  • Collects metrics for all running containers.
  • Collects events.
  • Collects process metrics.
  • Collects host logs.
  • Flexible source pattern allows you to specify field extraction rules.
  • Pre-built dashboards allow you to monitor your applications right away.

Collector without privileged flag

We use privileged in our examples because it is the easiest way to make it work out of the box on most distributions of Linux. Privileged is not required in most installations.

To be able to read IO information from /proc filesystem we need capability CAP_SYS_PTRACE, which sometimes can be protected by apparmor or similar. Instead of --privileged you can try to run --cap-add=SYS_PTRACE --security-opt apparmor:unconfined.


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which gives you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which helps developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.