Outcold Solutions LLC

Monitoring Docker Installation

With our solution for Monitoring Docker, you can start monitoring your clusters in under 10 minutes, including forwarding metadata-enriched container logs, host logs, and metrics.

Features:

  • Supports Docker versions starting from v1.9 (talk to us if you need support for earlier version).
  • Log forwarding is built on native JSON logging driver.
  • Tiny image, tiny binary. Very low memory, CPU, and disk consumption.
  • Logs are enriched with Docker metadata (container, image, labels, etc.).
  • Collects stats and events, allowing you to correlate logs with metrics.
  • Collects process metrics.
  • Security monitoring (privileged containers and enabled capabilities).
  • Multi-line events support.
  • Limits monitoring (CPU Shares, CPU Quotas, Memory Limits for containers).
  • Host logs collection.
  • Uses HTTP Event Collector to ingest data in Splunk. Requires Splunk version 6.5 or above (talk to us if you need support for an earlier version of Splunk).

  • At least once delivery guarantee.

Splunk configuration

Install Monitoring Docker application

Install Monitoring Docker from splunkbase. You need to install it on Search Heads only.

Enable HTTP Event Collector in Splunk

Outcold Solutions' Collector sends data to Splunk using HTTP Event Collector. By default, Splunk does not enable HTTP Event Collector. Please read HTTP Event Collector walkthrough to learn more about HTTP Event Collector.

After enabling HTTP Event Collector, you need to find correct Url for HTTP Event Collector and generate an HTTP Event Collector Token. If you are running your Splunk instance on hostname hec.example.com, it listens on port 8088, using SSL and token is B5A79AAD-D822-46CC-80D1-819F80D7BFB0 you can test it with the curl command as in the example below.

$ curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}

-k is necessary for self-signed certificates.

Install Collector for Docker

Pre-requirements

Some linux distributions, like CentOS by default enable journald logging driver instead of default JSON logging driver. You can verify which driver is used by default

$ docker info | grep "Logging Driver"
Logging Driver: json-file

If you see that default Logging Driver not the json-file you need to change it back to json-file.

As an example, if docker configuration file is under /etc/sysconfig/docker you can change the configuration and restart docker daemon after that with following commands.

$ sed -i 's/--log-driver=journald/--log-driver=json-file --log-opt max-size=1M --log-opt max-file=3/' /etc/sysconfig/docker
$ systemctl restart docker

Please follow the manual to learn how to configure default logging driver for containers:

Installation

Pull the latest version of the collector.

docker pull outcoldsolutions/collectorfordocker:3.0.93.180531

We recommend sticking to specific latest version to make the upgrade process more straightforward. Follow us on blog, twitter or subscribe to the newsletter to keep up to date with the releases.

Run collector image as in the example (command is using the same configuration as curl command above). Modify Splunk URL and Token value, and accept the license agreement.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
docker run -d \
    --name collectorfordocker \
    --volume /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro \
    --volume /proc:/rootfs/proc:ro \
    --volume /var/log:/rootfs/var/log:ro \
    --volume /var/lib/docker/containers/:/var/lib/docker/containers/:ro \
    --volume /var/run/docker.sock:/var/run/docker.sock:ro \
    --volume collector_data:/data/ \
    --cpus=1 \
    --cpu-shares=102 \
    --memory=256M \
    --restart=always \
    --env "COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0" \
    --env "COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0"  \
    --env "COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true"  \
    --env "COLLECTOR__EULA=general__acceptEULA=true" \
    --privileged \
    outcoldsolutions/collectorfordocker:3.0.93.180531

In case of AWS ECS you need to change /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro to /cgroup:/rootfs/sys/fs/cgroup:ro as ECS optimized images mount cgroup filesystem in root folder directly. See Monitoring Amazon Elastic Container Service Clusters in Splunk.

If you are using Docker Compose, use our docker-compose.yaml as a reference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
version: "3"
services:

  collectorfordocker:
    image: outcoldsolutions/collectorfordocker:3.0.93.180531
    volumes:
      - /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro
      - /proc:/rootfs/proc:ro
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/containers/:/var/lib/docker/containers/:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - collector_data:/data/
    environment:
      - COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0
      - COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0
      - COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true
      - COLLECTOR__EULA=general__acceptEULA=true
    restart: always
    deploy:
      mode: global
      restart_policy:
        condition: any
      resources:
        limits:
          cpus: '1'
          memory: 256M
        reservations:
          cpus: '0.05'
          memory: 64M
    privileged: true

volumes:
  collector_data:

If you use Splunk generated certificate, you probably want to add some SSL specific configuration. The easiest to get started with is --env "COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true" to skip SSL validation, as we specified in examples above.

Important note, that collector does not require you to change the default logging driver. It forwards logs from default JSON logging driver.

Give it a few moments to download the image and start the container. After the container is deployed, go to the Monitoring Kubernetes application in Splunk and you should see data on dashboards.

Deploying on Docker Swarm

To deploy on Docker Swarm you can use docker-compose.yaml file from above and use the the stack deploy command.

docker stack deploy --compose-file ./docker-compose.yaml collectorfordocker

Docker configuration

With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space. You can specify --log-driver=json-file --log-opt=max-size=100m --log-opt=max-file=5 for docker daemon. See Configure and troubleshoot the Docker daemon for more details.

Screencasts

Installation process

Solution Overview


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which gives you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which helps developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.