Metrics index

Collectord can send the stats it collects to a Splunk metrics index instead of, or beside, the events index. Use this page to decide which index fits your cluster, and to move your stats over.

Monitoring OpenShift has a second set of dashboards that read the metrics index with mstats. Every Alert Catalog template that reads stats has a metrics-index body, so a cluster that sends its stats there is fully covered.

Events or metrics index

Both work, and you can change your mind later. What differs is cost and speed. The figures below come from a Kubernetes cluster with Splunk 10.4, since Collectord sends OpenShift’s stats in the same records. Measured there, the same samples sent to a metrics index:

• cost about 1.5 times the license: Splunk licenses a metrics record at up to 150 bytes, while Collectord’s events-index stats records are compact lines;
• take about half the disk;
• aggregate 4 to 7 times faster over the same range, which shows on long ranges and large clusters.

The license cost differs by stream: process stats cost slightly less license as metrics, network stats several times more.

• Stay on the events index if license volume is what you watch most.
• Move to the metrics index if your searches over long ranges are slow, or you want the stats in Splunk’s Analytics Workspace.

The Splunk Usage Report in the app shows what each index costs you.

What goes where

[general] defaultIndexType sets where every one of those inputs sends its records. An input’s own indexType overrides it for that input. Each takes:

• events (the default) - the events-index records the app has always read.
• metrics - Splunk metrics-index records instead.
• both - the two at once. That is how you move without a gap: send both, compare the two sets of dashboards, then switch to metrics.

Send stats to the metrics index

1. In Splunk, create a metrics index, for example openshift_metrics (Settings › Indexes › New Index, Index Data Type: Metrics).
2. Allow your HTTP Event Collector token to write to it: add the metrics index to the token’s Allowed Indexes beside the events index. A metrics record that names no index goes to the token’s default index; when that is an events index, the record is indexed as an event and its measurements are lost, which is why step 3 names the metrics index.
3. Switch the inputs and name the metrics index. With Helm:

1collectord:
2  configuration:
3    general:
4      defaultIndexType: metrics
5    outputs:
6      splunk:
7        default:
8          defaultMetricsIndex: openshift_metrics

With the manifest, set defaultIndexType = metrics under [general] and defaultMetricsIndex = openshift_metrics under [output.splunk] in 001-general.conf.

That switches every input in the table above, the Prometheus inputs the control-plane dashboards read included. To keep one input on the events index, set its own indexType = events. A pod’s prometheus.<N>-indexType annotation does the same for its endpoint (see Prometheus metrics).

An input can also name its own index with metricsIndex, and send its metrics through another Splunk output with metricsOutput, bound to a token allowed to write that index.

4. Check the setup from a Collectord pod: collectord verify posts a test record to the metrics index and reports it (see Troubleshooting).

Choose the metrics-index dashboards

1. Open Setup › Configuration in the app.
2. Under Dashboards data source, choose Metrics-index dashboards and click Apply.

The app menu switches to the metrics-index set. Both sets stay installed, so you can switch back at any time.

The metrics-index dashboards and alerts read the metrics indexes through macro_openshift_metrics_base. It is (index=*) by default, every metrics index your role can search, so a metrics index with any name works. To read only yours, narrow the macro from the Macros panel of the same page, for example to (index=openshift_metrics).

Explore the metrics

In Splunk’s Analytics Workspace, the metrics show up under openshift.host.*, openshift.pod.*, openshift.container.*, openshift.process.*, openshift.net.*, openshift.mount.* and openshift.disk.*. Their dimensions are named the way the events index names its fields, with dots: openshift.namespace, openshift.pod.name, openshift.container.name. Prometheus metrics keep their own names, with each label as metric.label.<label>.