Metrics index - Monitoring Docker

Available since Collectord version 26.10

Collectord can send the stats it collects on a Docker host to a Splunk metrics index instead of, or beside, the events index. Use this page to decide which index fits your hosts, and to move your stats over.

Monitoring Docker has a second set of dashboards that read the metrics index with mstats. Every Alert Catalog template that reads stats has a metrics-index body, so a host that sends its stats there is fully covered.

Events or metrics index

Both work, and you can change your mind later. What differs is cost and speed. Splunk licenses a metrics record at 18 bytes per measurement, up to 150 bytes, while Collectord’s events-index stats records are compact lines.

The figures below come from a Kubernetes cluster with Splunk 10.4, not from Docker, so read them as a direction. Measured there, the same samples sent to a metrics index:

cost about 1.5 times the license;
take about half the disk;
aggregate 4 to 7 times faster over the same range, which shows on long ranges and many hosts.

The license cost differs by stream: process stats cost slightly less license as metrics, network stats several times more.

Stay on the events index if license volume is what you watch most.
Move to the metrics index if your searches over long ranges are slow, or you want the stats in Splunk’s Analytics Workspace.

The Splunk Usage Report in the app shows what each index costs you.

What goes where

Data	Events index	Metrics index
Host and container stats (`[input.system_stats]`)	yes	with `indexType`
Process stats (`[input.proc_stats]`)	yes	with `indexType`
Network interfaces (`[input.net_stats]`)	yes	with `indexType`
Filesystems (`[input.mount_stats]`)	yes	with `indexType`
Block devices (`[input.disk_stats]`)	yes	with `indexType`
Prometheus metrics (`[input.prometheus_auto]`, `[input.prometheus::<name>]`)	yes	with `indexType`
Collectord’s own metrics (`[input.collectord_metrics]`)	yes	with `indexType`
Network sockets, Docker events, objects, logs	yes	no

[general] defaultIndexType sets where every one of those inputs sends its records. An input’s own indexType overrides it for that input. Each takes:

events (the default) - the events-index records the app has always read.
metrics - Splunk metrics-index records instead.
both - the two at once. That is how you move without a gap: send both, compare the two sets of dashboards, then switch the inputs to metrics.

Send stats to the metrics index

In Splunk, create a metrics index, for example docker_metrics (Settings › Indexes › New Index, Index Data Type: Metrics).
Allow your HTTP Event Collector token to write to it: add the metrics index to the token’s Allowed Indexes beside the events index. A metrics record that names no index goes to the token’s default index; when that is an events index, the record is indexed as an event and its measurements are lost, which is why step 3 names the metrics index.
Name the metrics index on the output and switch the inputs. Add these lines to the docker run command from the installation guide, and re-create the container with them:

bash

1    --env "COLLECTOR__INDEXTYPE=general__defaultIndexType=metrics" \
2    --env "COLLECTOR__METRICSINDEX=output.splunk__defaultMetricsIndex=docker_metrics" \

The name after COLLECTOR__ is yours to pick, but each line needs its own: a repeated name keeps only the last value.

Other ways of running Collectord take the same values:

Docker Compose. In docker-compose.yaml, they go in the service’s environment: list (- COLLECTOR__METRICSINDEX=output.splunk__defaultMetricsIndex=docker_metrics).
Quadlet. In a Quadlet unit, they go on Environment= lines.
Your own image (Configuration). Set defaultIndexType = metrics under [general] and defaultMetricsIndex = docker_metrics under [output.splunk] of its .conf file instead.

defaultIndexType switches every input in the table above, the Prometheus inputs included. To keep one input on the events index, set its own indexType = events. A container’s collectord.io/prometheus.<N>-indexType label does the same for that container’s endpoint (see annotations).

An input can also name its own index with metricsIndex, and send its metrics through another Splunk output with metricsOutput, bound to a token allowed to write that index.

Check the setup: docker exec collectorfordocker /collectord verify posts a test record to the metrics index and reports it (see Troubleshooting).

Choose the metrics-index dashboards

Open Setup › Configuration in the app.
Under Dashboards data source, choose Metrics-index dashboards and click Apply.

The app menu switches to the metrics-index set. Both sets stay installed, so you can switch back at any time. Every page that reads stats has a twin there (hosts_m, container_m, process_m and the rest). Events, Logs and Network read the events index and appear in both menus.

The metrics-index dashboards and alerts read the metrics indexes through macro_docker_metrics_base. It is (index=*) by default, every metrics index your role can search, so a metrics index with any name works. To read only yours, narrow the macro from the Macros panel of the same page, for example to (index=docker_metrics).

Two tiles at the bottom of the page show whether stats arrive where you expect: measurements in the metrics indexes the macro reads, and stats records that reached the events index.

The metrics-index dashboards filter and group by cluster through the docker.cluster dimension. Collectord sends it on every metrics-index record from [general] fields.docker_cluster, the COLLECTOR__CLUSTER line of the installation commands. A host without it shows -.

The events-index dashboards can also take a cluster from a Docker engine’s cluster= label or an ECS cluster label; the metrics-index ones can’t. To see each cluster the same way in both sets, name it with fields.docker_cluster (see Multiple clusters).

Explore the metrics

In Splunk’s Analytics Workspace, the metrics show up under docker.host.*, docker.container.*, docker.process.*, docker.net.*, docker.mount.* and docker.disk.*, with CPU in cores (docker.container.cpu.total_cores). Their dimensions are named the way the events index names its fields, with dots: docker.cluster, docker.container.id, docker.container.name. A container’s labels ride on its records as the multivalue docker.container.labels.

Prometheus metrics keep their own names, with each label as metric.label.<label>, and one record packs every measurement of a label set.

Before 26.10, each Prometheus sample sent to a metrics index was a record of its own, the labels were metric_label_<label> dimensions, and the records carried metric_type and metric_help. A search written against those dimensions returns nothing for the records sent after the upgrade until it uses the dotted names.