When you want to alert on Docker configuration drift — a container running with an unexpected image, an environment variable that shouldn’t be there, an image built without the labels your team requires — you need the full container and image specs in Splunk, not just logs and metrics. Starting with version 5.9, Collectord can poll the Docker API and forward those objects on a schedule.
Configuration
Collectord ships with two pre-configured polls — one for containers, one for images — both disabled by default. Enable the ones you need.
1[input.docker_api::containers]
2
3# disable docker events
4disabled = true
5
6path = /containers/json
7inspectPath = /containers/{{.Id}}/json
8interval = 5m
9query = all=1
10apiVersion =
11
12# override type
13type = docker_objects
14
15# specify Splunk index
16index =
17
18# set output (splunk or devnull, default is [general]defaultOutput)
19output =
20
21
22[input.docker_api::images]
23
24# disable docker events
25disabled = true
26
27path = /images/json
28inspectPath = /images/{{.Id}}/json
29interval = 5m
30query = all=0
31apiVersion =
32
33# override type
34type = docker_objects
35
36# specify Splunk index
37index =
38
39# set output (splunk or devnull, default is [general]defaultOutput)
40output =The simplest way to flip these on is with environment variables on the collectorfordocker container:
1 ...
2 --env "COLLECTOR__DOCKER_CONTAINERS=input.docker_api::containers__disabled=false" \
3 --env "COLLECTOR__DOCKER_IMAGES=input.docker_api::images__disabled=false" \
4 ...Searching the data
With the configuration above, every object is re-sent every 5 minutes. If you’re using join or populating lookups, your search window needs to be longer than interval — 6 minutes is a safe default for a 5-minute poll.
The source name
Each event’s source follows the format /docker/{key}, where key is the suffix of the stanza name — for example, [input.docker_api::containers] produces /docker/containers.
Event format
Containers are forwarded as returned by /containers/{id}/json, and images as returned by /images/{id}/json. See the Engine API for the full schema.
Searching the data
Because the same object can show up multiple times in the same window — once per modification, plus the periodic poll — group by the object’s unique identifier and take the latest version per group:
1sourcetype="docker_objects" source="/docker/containers" |
2stats latest(_raw) as _raw by Id |
3spath output=Created path=Created |
4spath output=Name path=Name |
5spath output=Status path=State.Status |
6table Created, Name, Status