Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring OpenShift

Collecting Prometheus metrics

Most of the components in the OpenShift control plane export metrics in Prometheus format. The collectord can read these metrics and forward them to Splunk Enterprise or Splunk Cloud. Our installation has default configurations for collecting metrics from the API Server, controllers, kubelets, and etcd cluster. In most OpenShift providers, you don’t need to do additional configuration to see these metrics.

If your applications export metrics in Prometheus format, you can also use our collectord to forward these metrics to Splunk Enterprise or Splunk Cloud.

Forwarding metrics from Pods

Please read our documentation on annotations, to learn how you can define forwarding metrics from Pods.

Defining prometheus input

We deploy the collectord in 3 different workloads. Depending on where you want to collect your metrics, you should plan to include your Prometheus metrics.

  • 002-daemonset.conf is installed on all nodes (masters and non-masters). Use this configuration if you need to collect metrics from all nodes from local ports. An example of these metrics is kubelet metrics.
  • 003-daemonset-master.conf is installed only on master nodes. Use this configuration to collect metrics only from master nodes from local ports. Examples of these metrics are control plane processes and etcd running on masters.
  • 004-addon.conf is installed as a deployment and used only once in the whole cluster. Place your Prometheus configuration here if you want to collect metrics from endpoints or services. Examples of these Prometheus configurations are the controller manager and scheduler, which can be accessed only from an internal network and can be discovered with endpoints. Another example is an etcd cluster running outside the OpenShift cluster.

Default configuration

Kubelet

On every node, the collectord reads and forwards kubelet metrics. We deploy this configuration in 002-daemonset.conf.

[input.prometheus::kubelet]

# disable prometheus kubelet metrics
disabled = false

# override type
type = prometheus

# specify Splunk index
index =

# Override host (environment variables are supported)
host = ${KUBERNETES_NODENAME}

# Override source
source = kubelet

# how often to collect prometheus metrics
interval = 60s

# prometheus endpoint
endpoint = https://127.0.0.1:10250/metrics

# token for "Authorization: Bearer $(cat tokenPath)"
tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token

# server certificate for certificate validation
certPath = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# client certificate for authentication
clientCertPath =

# Allow invalid SSL server certificate
insecure = true

# include metrics help with the events
# can be useful to explore prometheus metrics
includeHelp = false

API Server

On master nodes, the collectord reads and forwards metrics from the API server. We deploy this configuration using 003-daemonset-master.conf.

[input.prometheus::kubernetes-api]

# disable prometheus kubernetes-api input
disabled = false

# override type
type = prometheus

# specify Splunk index
index =

# override host
host = ${KUBERNETES_NODENAME}

# override source
source = kubernetes-api

# how often to collect prometheus metrics
interval = 60s

# prometheus endpoint
# at first trying to get it from localhost (that way avoiding load balancer, if multiple)
# as fallback using proxy
endpoint.1localhost = https://127.0.0.1:8443/metrics
endpoint.2kubeapi = https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/metrics

# token for "Authorization: Bearer $(cat tokenPath)"
tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token

# server certificate for certificate validation
certPath = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# client certificate for authentication
clientCertPath =

# Allow invalid SSL server certificate
insecure = true

# include metrics help with the events
includeHelp = false

Controller

On master nodes, the collectord reads and forwards metrics from the controller. We deploy this configuration using 003-daemonset-master.conf.

[input.prometheus::controller]

# disable prometheus controller metrics
disabled = false

# override type
type = prometheus

# specify Splunk index
index =

# override host
host = ${KUBERNETES_NODENAME}

# override source
source = controller

# how often to collect prometheus metrics
interval = 60s

# prometheus endpoint
endpoint.https = https://127.0.0.1:8444/metrics

# token for "Authorization: Bearer $(cat tokenPath)"
tokenPath = /var/run/secrets/kubernetes.io/serviceaccount/token

# server certificate for certificate validation
certPath =

# client certificate for authentication
clientCertPath =
clientKeyPath =

# Allow invalid SSL server certificate
insecure = true

# include metrics help with the events
includeHelp = false

etcd

On master nodes, the collectord reads and forwards metrics from etcd processes. We deploy this configuration using 003-daemonset-master.conf.

[input.prometheus::etcd]

# disable prometheus etcd metrics
disabled = false

# override type
type = prometheus

# specify Splunk index
index =

# override host
host = ${KUBERNETES_NODENAME}

# override source
source = etcd

# how often to collect prometheus metricd
interval = 60s

# prometheus endpoint
endpoint.https = https://:2379/metrics

# token for "Authorization: Bearer $(cat tokenPath)"
tokenPath =

# server certificate for certificate validation
certPath = /rootfs/etc/origin/master/master.etcd-ca.crt

# client certificate for authentication
clientCertPath = /rootfs/etc/origin/master/master.etcd-client.crt
clientKeyPath = /rootfs/etc/origin/master/master.etcd-client.key

# Allow invalid SSL server certificate
insecure = true

# include metrics help with the events
includeHelp = false

This configuration works when you run an etcd cluster with master nodes. With this configuration, the collectord tries to collect metrics using the http scheme first and https after that. For https, the collectord uses certPath, clientCertPath, and clientKeyPath, which are mounted from the host.

...
  volumeMounts:
  ...
  - name: origin-certs
    mountPath: /rootfs/etc/origin/master/
    readOnly: true
...
volumes:
- name: origin-certs
  hostPath:
    path: /etc/origin/master/

Verify that these certificates are available; if not, make appropriate changes.

If your etcd cluster is a dedicated set of nodes, you can define Prometheus collection in 004-addon.conf.

Metrics format (Splunk Index Type = Events)

Prometheus defines several types of metrics.

Each metric value in Splunk has fields:

  • metric_type - one of the types from the Prometheus metric types.
  • metric_name - the name of the metric.
  • metric_help - only if includeHelp is set to true, you will see definition of this metric.
  • metric_label_XXX - if the metric has labels, you will be able to see them attached to the metric values.
  • seed - unique value from the host for specific metric collection.

Based on the metric type you can find various values for the metrics.

  • counter

    • v - current counter value
    • d - the difference with a previous value
    • s - period for which this difference is calculated (in seconds)
    • p - (deprecated) period for which this difference is calculated (in nanoseconds)

  • summary and histogram

    • v - value
    • c - counter specified for this summary or histogram metric

  • All others

    • v - value

If you have specified to include help with the metrics, you can explore all available metrics with this search.

sourcetype="prometheus"
|  stats latest(_raw) by source, metric_type, metric_name, metric_help

Explore Prometheus Metrics

Metrics format (Splunk Index Type = Metrics)

Using Collectord version 5.24+, you can configure Prometheus metrics to be forwarded to the Splunk Metrics Index by configuring indexType = metrics in the ConfigMap under the [input.prometheus::X] stanza, or by using annotations like collectord.io/prometheus.1-indexType=metrics.

The values of Prometheus metrics are sent as metric values, and additional labels are attached as metric_label_XXX fields.

You can leverage the Splunk Analytics dashboard to explore the metrics.

When you decide to configure Prometheus metrics to be sent to the Splunk Metrics Index, we suggest defining an additional Splunk Output that will forward to the token that is set by default to be forwarded to Metrics Indexes (by default and additional indexes).

Explore Prometheus Metrics

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.