Metrics index

Collectord can send the stats it collects on a Linux host to a Splunk metrics index, instead of or beside the events index. Monitoring Linux fully covers a host that sends its stats there.

The app has a second set of dashboards that read the metrics index with mstats, and every Alert Catalog template that reads stats has a metrics-index body.

What goes where

Logs always go to the events index. [general] defaultIndexType sets where every stats input sends its records, and an input’s own indexType overrides it for that input. Each takes:

• events (the default): the events-index records the app has always read.
• metrics: Splunk metrics-index records instead.
• both: the two at once, which is how you compare the two sets of dashboards before you switch.

Send stats to the metrics index

1. In Splunk, create a metrics index, for example linux_metrics (Settings › Indexes › New Index, Index Data Type: Metrics).
2. Allow your HTTP Event Collector token to write to it: add the metrics index to the token’s Allowed Indexes beside the events index. A metrics record sent through a token whose index is an events one is indexed as an event whose text is the word metric, and every measurement in it is lost.
3. In /opt/collectorforlinux/etc/002-user.conf, switch the inputs and name the metrics index. The 002-user.conf the packages install, and the tarball’s 002-user.conf.example, carry this block commented out:

1[general]
2defaultIndexType = metrics
3
4[output.splunk]
5defaultMetricsIndex = linux_metrics

To keep one input on the events index, set its own indexType = events. A stats input can also name its own index with metricsIndex, and send its metrics-index records through another Splunk output with metricsOutput: an [output.splunk::<name>] section bound to a token allowed to write that index.

4. Restart the service and check the setup:

1sudo systemctl restart collectorforlinux
2sudo /opt/collectorforlinux/bin/collectord verify

collectord verify posts a test record to the metrics index and reports it. It also refuses a configuration that cannot do what it says, such as a metricsIndex on an input whose indexType is still events, and names the section.

Choose the metrics-index dashboards

1. Open Setup › Configuration in the app.
2. Under Dashboards data source, choose Metrics-index dashboards and click Apply.

The app menu switches to the metrics-index set (hosts_m, host_m, process_m, storage_m, diskstats_m, collectord_metrics_m and the rest). Both sets stay installed, so you can switch back at any time. On a new install, the page recommends the set that received data in the last 15 minutes.

The metrics-index dashboards and alerts read every metrics index your role can search. They go through macro_linux_metrics_base, (index=*) by default, so a metrics index with any name works without changing anything. Every search also names the app’s source types, so another product’s metrics in the same index stay out.

To read only yours, narrow the macro, for example to (index=linux_metrics), from the Macros panel of the same page. The panel opens the macro’s editor in Settings › Advanced search › Search macros, which needs write access to the app (the admin, power or sc_admin role) and the app’s setup finished.

Two tiles at the bottom of the page show whether stats arrive where you expect: measurements in the metrics indexes the macro reads, and stats records that reached the events index.

Explore the metrics

The metrics show up in Splunk’s Analytics Workspace under linux.host.*, linux.process.*, linux.net.*, linux.mount.* and linux.disk.*. They carry dimensions to split by:

• host and linux.cluster on every metric;
• device on network interfaces and block devices;
• linux.mount.path on filesystems;
• pid, comm, state, systemd_unit and more on processes.

Collectord’s own metrics keep their names (splunk_post_bytes_sum, pipelines_num) with source=collectord:<instance>.

License

Splunk licenses a metrics-index record at up to 150 bytes, however many measurements it carries. collectorforlinux’s events-index stats records are compact lines. Measured on Splunk 10.4.3, per record:

So moving the stats to a metrics index does not lower the licensed volume, except for process stats. Choose it for what mstats gives you: faster searches over long ranges, and the Analytics Workspace. To see what each index costs on your license, use the Splunk Usage Report in the app.