By default collectord for docker does not collect any metrics in Prometheus format. But it has capabilities to do so.

Configuration

The generic Prometheus configuration section is

 1[input.prometheus::ANYNAME]
 2
 3# disable input
 4disabled = false
 5
 6# specify the type for current input
 7type = docker_prometheus
 8
 9# specify Splunk index for current input
10index =
11
12# specify the host for current input
13host =
14
15# specify the source
16source =
17
18# how often to collect prometheus metrics
19interval = 60s
20
21# Prometheus endpoint, multiple values can be specified, collectord tries them in order till finding the first
22# working endpoint.
23# At first trying to get it through proxy
24endpoint.1 = 
25# endpoint.2 =
26
27# token for "Authorization: Bearer $(cat tokenPath)"
28tokenPath =
29
30# server certificate for certificate validation
31certPath =
32
33# client certificate and key for authentication
34clientCertPath =
35clientKeyPath =
36
37# Allow invalid SSL server certificate
38insecure = false
39
40# include metrics help with the events
41includeHelp = false

You can find variety of the services, exporting metrics in Prometheus format.

Example. Collecting metrics from nginx service.

For this example we will use sophos/nginx-prometheus-metrics image.

The image sophos/nginx-prometheus-metrics is not an official nginx image. We use it only for demo purposes.

We use a command from the image example to run it

1docker run -d --rm -it -p 80:80 -p 1314:1314 -p 9527:9527 sophos/nginx-prometheus-metrics

For this input we define configuration with 002.conf. The most important part of configuration is to specify how collector can get to the endpoint with metrics. We use simple approach by publishing port in outside world, and accessing it with the hostname of the host.

 1[general]
 2acceptLicense = true
 3
 4[output.splunk]
 5url = https://hec.example.com:8088/services/collector/event/1.0
 6token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
 7insecure = true
 8
 9[input.prometheus::nginx-example]
10disabled = false
11type = docker_prometheus
12index =
13host =
14source = docker_nginx_example
15interval = 60s
16endpoint.1 = http://host.docker.internal:9527/metrics
17tokenPath =
18certPath =
19clientCertPath =
20clientKeyPath =
21insecure = false
22includeHelp = true

Build the image with the embedded configuration.

1FROM outcoldsolutions/collectorfordocker:25.10.3
2
3COPY 002.conf /config/002.conf

Run it with provided installation guidance.

Metrics format

Prometheus defines several types of metrics.

Each metric value in Splunk has fields:

metric_type - one of the types from the Prometheus metric types.
metric_name - the name of the metric.
metric_help - only if includeHelp is set to true<, you will see definition of this metric.
metric_label_XXX - if the metric has labels, you will be able to see them attached to the metric values.
seed - unique value from the host for specific metric collection.

Based on the metric type you can find various values for the metrics.

counter
- v - current counter value
- d - the difference with a previous value
- p - period for which this difference is calculated (in nanoseconds)
summary and histogram
- v - value
- c - counter specified for this summary or histogram metric
All others
- v - value

If you have specified to include help with the metrics, you can explore all available metrics with the search.

1sourcetype="docker_prometheus"
2|  stats latest(_raw) by source, metric_type, metric_name, metric_help

Monitoring Docker

Splunk HTTP Event Collector

Configuration

Example. Collecting metrics from nginx service.

Metrics format

Links

About Outcold Solutions