Most of the components in the OpenShift control plane export metrics in Prometheus format. The collectord can read these metrics and forward them to Splunk Enterprise or Splunk Cloud. Our installation has default configurations for collecting metrics from the API Server, controllers, kubelets, and etcd cluster. In most OpenShift providers, you don’t need to do additional configuration to see these metrics.

If your applications export metrics in Prometheus format, you can also use our collectord to forward these metrics to Splunk Enterprise or Splunk Cloud.

Forwarding metrics from Pods

Please read our documentation on annotations, to learn how you can define forwarding metrics from Pods.

Defining prometheus input

We deploy the collectord in 3 different workloads. Depending on where you want to collect your metrics, you should plan to include your Prometheus metrics.

002-daemonset.conf is installed on all nodes (masters and non-masters). Use this configuration if you need to collect metrics from all nodes from local ports. An example of these metrics is kubelet metrics.
003-daemonset-master.conf is installed only on master nodes. Use this configuration to collect metrics only from master nodes from local ports. Examples of these metrics are control plane processes and etcd running on masters.
004-addon.conf is installed as a deployment and used only once in the whole cluster. Place your Prometheus configuration here if you want to collect metrics from endpoints or services. Examples of these Prometheus configurations are the controller manager and scheduler, which can be accessed only from an internal network and can be discovered with endpoints. Another example is an etcd cluster running outside the OpenShift cluster.

Default configuration

Kubelet

On every node, the collectord reads and forwards kubelet metrics. We deploy this configuration in 002-daemonset.conf.

 1[input.prometheus::kubelet]
 2
 3# disable prometheus kubelet metrics
 4disabled = false
 5
 6# override type
 7type = prometheus
 8
 9# specify Splunk index
10index =
11
12# Override host (environment variables are supported)
13host = ${KUBERNETES_NODENAME}
14
15# Override source
16source = kubelet
17
18# how often to collect prometheus metrics
19interval = 60s
20
21# prometheus endpoint
22endpoint = https://127.0.0.1:10250/metrics
23
24# token for "Authorization: Bearer $(cat tokenPath)"
25tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token
26
27# server certificate for certificate validation
28certPath = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
29
30# client certificate for authentication
31clientCertPath =
32
33# Allow invalid SSL server certificate
34insecure = true
35
36# include metrics help with the events
37# can be useful to explore prometheus metrics
38includeHelp = false

API Server

On master nodes, the collectord reads and forwards metrics from the API server. We deploy this configuration using 003-daemonset-master.conf.

 1[input.prometheus::kubernetes-api]
 2
 3# disable prometheus kubernetes-api input
 4disabled = false
 5
 6# override type
 7type = prometheus
 8
 9# specify Splunk index
10index =
11
12# override host
13host = ${KUBERNETES_NODENAME}
14
15# override source
16source = kubernetes-api
17
18# how often to collect prometheus metrics
19interval = 60s
20
21# prometheus endpoint
22# at first trying to get it from localhost (that way avoiding load balancer, if multiple)
23# as fallback using proxy
24endpoint.1localhost = https://127.0.0.1:8443/metrics
25endpoint.2kubeapi = https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/metrics
26
27# token for "Authorization: Bearer $(cat tokenPath)"
28tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token
29
30# server certificate for certificate validation
31certPath = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
32
33# client certificate for authentication
34clientCertPath =
35
36# Allow invalid SSL server certificate
37insecure = true
38
39# include metrics help with the events
40includeHelp = false

Controller

On master nodes, the collectord reads and forwards metrics from the controller. We deploy this configuration using 003-daemonset-master.conf.

 1[input.prometheus::controller]
 2
 3# disable prometheus controller metrics
 4disabled = false
 5
 6# override type
 7type = prometheus
 8
 9# specify Splunk index
10index =
11
12# override host
13host = ${KUBERNETES_NODENAME}
14
15# override source
16source = controller
17
18# how often to collect prometheus metrics
19interval = 60s
20
21# prometheus endpoint
22endpoint.https = https://127.0.0.1:8444/metrics
23
24# token for "Authorization: Bearer $(cat tokenPath)"
25tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token
26
27# server certificate for certificate validation
28certPath =
29
30# client certificate for authentication
31clientCertPath =
32clientKeyPath =
33
34# Allow invalid SSL server certificate
35insecure = true
36
37# include metrics help with the events
38includeHelp = false

etcd

On master nodes, the collectord reads and forwards metrics from etcd processes. We deploy this configuration using 003-daemonset-master.conf.

 1[input.prometheus::etcd]
 2
 3# disable prometheus etcd metrics
 4disabled = false
 5
 6# override type
 7type = prometheus
 8
 9# specify Splunk index
10index =
11
12# override host
13host = ${KUBERNETES_NODENAME}
14
15# override source
16source = etcd
17
18# how often to collect prometheus metricd
19interval = 60s
20
21# prometheus endpoint
22endpoint.https = https://:2379/metrics
23
24# token for "Authorization: Bearer $(cat tokenPath)"
25tokenPath =
26
27# server certificate for certificate validation
28certPath = /rootfs/etc/origin/master/master.etcd-ca.crt
29
30# client certificate for authentication
31clientCertPath = /rootfs/etc/origin/master/master.etcd-client.crt
32clientKeyPath = /rootfs/etc/origin/master/master.etcd-client.key
33
34# Allow invalid SSL server certificate
35insecure = true
36
37# include metrics help with the events
38includeHelp = false

This configuration works when you run an etcd cluster with master nodes. With this configuration, the collectord tries to collect metrics using the http scheme first and https after that. For https, the collectord uses certPath, clientCertPath, and clientKeyPath, which are mounted from the host.

 1...
 2  volumeMounts:
 3  ...
 4  - name: origin-certs
 5    mountPath: /rootfs/etc/origin/master/
 6    readOnly: true
 7...
 8volumes:
 9- name: origin-certs
10  hostPath:
11    path: /etc/origin/master/

Verify that these certificates are available; if not, make appropriate changes.

If your etcd cluster is a dedicated set of nodes, you can define Prometheus collection in 004-addon.conf.

Metrics format (Splunk Index Type = Events)

Prometheus defines several types of metrics.

Each metric value in Splunk has fields:

metric_type - one of the types from the Prometheus metric types.
metric_name - the name of the metric.
metric_help - only if includeHelp is set to true, you will see definition of this metric.
metric_label_XXX - if the metric has labels, you will be able to see them attached to the metric values.
seed - unique value from the host for specific metric collection.

Based on the metric type you can find various values for the metrics.

counter
- v - current counter value
- d - the difference with a previous value
- s - period for which this difference is calculated (in seconds)
- p - (deprecated) period for which this difference is calculated (in nanoseconds)
summary and histogram
- v - value
- c - counter specified for this summary or histogram metric
All others
- v - value

If you have specified to include help with the metrics, you can explore all available metrics with this search.

1sourcetype="prometheus"
2|  stats latest(_raw) by source, metric_type, metric_name, metric_help

Explore Prometheus Metrics

Metrics format (Splunk Index Type = Metrics)

Using Collectord version 5.24+, you can configure Prometheus metrics to be forwarded to the Splunk Metrics Index by configuring indexType = metrics in the ConfigMap under the [input.prometheus::X] stanza, or by using annotations like collectord.io/prometheus.1-indexType=metrics.

The values of Prometheus metrics are sent as metric values, and additional labels are attached as metric_label_XXX fields.

You can leverage the Splunk Analytics dashboard to explore the metrics.

When you decide to configure Prometheus metrics to be sent to the Splunk Metrics Index, we suggest defining an additional Splunk Output that will forward to the token that is set by default to be forwarded to Metrics Indexes (by default and additional indexes).