Concepts

This page shows how Monitoring Docker works: its two pieces, what Collectord forwards, and where a setting comes from. Read it once, and the rest of the docs make sense without piecing it together from installation, configuration and labels.

Two pieces, one product

The product ships in two halves that share a version number:

• Collectord is the agent. It runs as a single container on each Docker host, watches the Docker daemon for new containers, and forwards their logs and metrics to Splunk over HEC.
• Monitoring Docker is the Splunk app: dashboards, alerts, search macros and field extractions. It reads what Collectord forwarded and doesn’t talk to the Docker daemon directly.

When you upgrade, update both to the same version. The release notes cover Splunk-app changes first, then Collectord updates:.

What Collectord forwards

Collectord covers a broad range of data sources, each configured in an [input.*] section. In Splunk, you see them in these categories:

• Logs. Container stdout / stderr, application logs from files inside the container or on a mounted volume, and host logs (the systemd journal and /var/log/*.log).
• Metrics. Host, container and process stats from cgroups and /proc; network and socket-table metrics; mount and disk metrics; Prometheus metrics scraped from your containers; and Collectord’s own internal metrics.
• Events. Docker daemon events.
• Objects. Containers, images and volumes, polled from the Docker API every 5 minutes, and a Swarm’s services, tasks and nodes, polled every minute. All of them are on by default: see Object polling.

Each input has its own sourcetype (docker_logs, docker_host_logs, docker_stats_v2_host, docker_stats_v2_cgroup, docker_proc_stats_v2, docker_net_stats_v2, docker_events, docker_objects, docker_prometheus, …) and a default index. The full list of inputs and types lives in the Configuration reference.

Inputs, outputs, and pipes

Internally, Collectord is a pipeline of three concepts:

• An input discovers data: [input.files] reads the containers’ log files, [input.files::logs] the host’s /var/log/*.log, [input.docker_events] watches the daemon, and [input.prometheus_auto] scrapes containers labeled with a metrics endpoint.
• A pipe transforms events on the way through: replace, hash, extract, override, sample, throttle.
• An output ships the result somewhere. [output.splunk] is the default, and devnull drops data. You can define several Splunk outputs and route different inputs to each.

Most container annotations just configure pipes and the destination output, scoped to one container.

Where configuration comes from

You can set the same setting in several places. They layer in this order, highest priority first:

1. Container labels. The most specific layer. Collectord accepts both the collectord.io/{label}={value} and the io.collectord.{label}={value} forms.
2. Environment variables on the Collectord container. --env COLLECTOR__X="<section>__<key>=<value>" overrides a single config value without rebuilding the image.
3. Mounted config files. Additional *.conf files mounted into /config/ on the Collectord container. Collectord reads everything matching /config/*.conf in alphabetical order, and later files override earlier ones.
4. Default 001-general.conf. The configuration shipped inside the Collectord image at /config/001-general.conf. The lowest-priority layer.

When a label doesn’t behave the way you expect, run collectord describe from inside the Collectord container: see Troubleshooting → Describe.

Auto-discovery vs explicit opt-in

Collectord finds some data on its own. For the rest, you point it there with a label or the configuration:

If something you expect isn’t in Splunk, first ask: was it auto-discovered, or do you need to point Collectord at it?

How data flows

Collectord runs on each Docker host and reads from local sources (Docker socket, filesystem, cgroups, /proc). It pushes events to your Splunk HTTP Event Collector, and the Splunk app on the search head reads them back through macros scoped to the right indexes:

1Your container ─┐
2                ├─► [input] ──► [pipe] ──► [output] ──► Splunk HEC ──► Splunk indexer
3Your label    ──┘                                                          │
4                                                                           ▼
5Splunk role  ◄── [Monitoring Docker app: dashboards, alerts, macros] ◄─────┘

Where to go next

• New host: Installation.
• Tuning what gets forwarded for a specific container: Container annotations.
• Splitting indexes by datatype, or routing to multiple Splunk clusters: Splunk indexes and Splunk HTTP Event Collector.
• Something in the dashboards looks wrong: Troubleshooting.