Metrics index - Monitoring macOS

Collectord for macOS sends every stat to a Splunk metrics index: what goes where, naming the index, the HEC token, and reading the metrics.

Collectord for macOS sends every stat it collects to a Splunk metrics index, and the logs and the other events to an events index, so a Mac sends to two indexes. Use this page to name the metrics index, check that the stats reach it, and search it.

Monitoring macOS reads every stat with mstats, and every Alert Catalog template over a stat reads the metrics index.

What goes where

Data	Index
Host stats (`[input.system_stats]`)	metrics
Hardware: CPU, GPU, power, thermals, fans, disks, filesystems, network (`[input.macos_stats]`)	metrics
Processes and virtual machines (`[input.macos_processes]`)	metrics
Posture, updates, battery, power state, users, launchd services, Xcode, compliance (`[input.macos_status]`)	metrics
Unified log counts and the noise sampler (`[input.macos_log_counts]`)	metrics
Collectord’s own metrics (`[input.collectord_metrics]`)	metrics
Prometheus scrapes (`[input.prometheus::<name>]`) and scripts with `format = prometheus`	metrics
Unified log entries and sample messages (`[input.macos_logs]`)	events
Host events and launchd transitions (`[input.macos_events]`)	events
Crash reports (`[input.macos_crash_reports]`)	events
Connections and listening sockets (`[input.macos_connections]`)	events
Applications (`[input.macos_apps]`)	events
The install log, Collectord’s log, your log files (`[input.files::<name>]`)	events

Name the metrics index

The 002-user.conf the package installs sends every stat to macos_metrics:

002-user.conf ini

1[output.splunk]
2defaultMetricsIndex = macos_metrics

To use another index, change it there, or set the MetricsIndex managed preference in your configuration profile. A stats input can also name its own index with metricsIndex. It can send its records through another Splunk output with metricsOutput: an [output.splunk::<name>] section bound to a token allowed to write that index.

Whatever its name, create the index as a metrics index (Settings › Indexes › New Index, Index Data Type: Metrics). Then add it to your HTTP Event Collector token’s Allowed Indexes.

sudo /opt/collectorformacos/bin/collectord verify posts a test record to every metrics index the configuration sends to, and reports whether the token may write it.

Warning
Without a metrics index named anywhere, the stats go to the token’s default index. That is normally an events index, where a metrics record is indexed as an event whose text is the word metric, and every measurement in it is lost.

How the app reads it

The dashboards and alerts read the metrics through macro_macos_metrics_base. It is (index=*) by default, every metrics index your role can search, so a metrics index with any name works without changing anything. Every search also names the app’s source types, so another product’s metrics in the same index stay out.

To read only yours, narrow the macro from the Macros panel of Setup › Configuration, for example to (index=macos_metrics).

Two tiles on the same page show whether the stats arrive where you expect:

the Macs with measurements in the metrics indexes the macro reads, in the last 15 minutes;
the stats records that reached an events index instead.

A number above 0 in the second means a token whose default index is an events index, or a metrics index the token may not write.

With field filters turned on ([search] field_filters in limits.conf), a field filter can make the dashboards fail (“Your search is restricted by a field filter”). That happens when the filter is on any metrics index macro_macos_metrics_base reaches, for a role the filter applies to that lacks the run_commands_ignoring_field_filter capability.
Limit the macro to unfiltered indexes: (index=macos_metrics), or (index=* NOT index=<filtered>). When your data’s index is the filtered one, grant those roles the capability or exempt them from the filter instead.

Explore the metrics

The metrics show up in Splunk’s Analytics Workspace under macos.*: macos.host.*, macos.cpu.*, macos.gpu.*, macos.power.*, macos.disk.*, macos.filesystem.*, macos.network.*, macos.process.*, macos.battery.*, macos.posture.* and more. Split them by these dimensions:

host, macos.cluster (the fleet label) and macos.hardware.class on every metric;
macos.process.name on processes;
macos.disk.name on disks;
macos.filesystem.mount on filesystems;
macos.network.interface on network interfaces;
the Mac’s model, chip and macOS version on the host record (macos.model, macos.chip, macos.os.version).

Collectord’s own metrics keep their names (splunk_post_bytes_sum, pipelines_num), with sourcetype=macos_prometheus.

For example, the CPU every fleet label used over the last hour, in cores:

text

1| mstats avg(macos.host.cpu.user_cores) as user avg(macos.host.cpu.system_cores) as system WHERE index=macos_metrics earliest=-1h BY macos.cluster span=5m

The Splunk Usage Report in the app shows what each index costs on your license, read from Splunk’s own license meter.