Monitoring Kubernetes

Splunk indexes

Split logs and metrics into separate Splunk indexes

Out of the box, collectorforkubernetes writes everything to the default index of your HTTP Event Collector token. This page shows how to send logs and metrics to indexes of your own, and how to make the Monitoring Kubernetes app search them.

Each token has an allow-list of indexes it can write to, and one of them is the default when the sender doesn’t pick one. The Monitoring Kubernetes app, in turn, assumes those indexes are searchable by default for your Splunk role: main is the obvious example.

If you forward to an index your role doesn’t search by default, the dashboards come up empty, even though Splunk indexes the data.

You have two ways to fix that. The first is to add the index to Indexes searched by default for your role, under Settings › Access Control › Roles:

Splunk - Indexes searched by default

The second is to update the search macros the app uses, so the dashboards search your indexes explicitly. Edit them in Splunk Web under Settings › Advanced search › Search macros. The Macros panel on the app’s Setup › Configuration page links the ones you are most likely to change. Or override $SPLUNK_HOME/etc/apps/monitoringkubernetes/default/macros.conf with a local/macros.conf:

Monitoring Kubernetes - Macros

Every events-index macro builds on a single base macro, macro_kubernetes_base, so you typically set the index list only once:

text
1macro_kubernetes_base = (index=kubernetes_stats OR index=kubernetes_logs)
Available since Collectord version 26.10

If Collectord sends stats to a Splunk metrics index, macro_kubernetes_metrics_base names it for the metrics-index dashboards and alerts. It is (index=*) by default: every metrics index your role can search. See Metrics index.

For finer-grained control, such as pinning a specific datatype to a specific index and sourcetype, override the individual macro instead:

text
1macro_kubernetes_stats_host = (index=kube_system_stats sourcetype=kubernetes_stats_v2_host)

The macros worth knowing about:

MacroWhat it reads
macro_kubernetes_eventsAll the Kubernetes events
macro_kubernetes_objectsThe API objects Collectord watches
macro_kubernetes_host_logsHost logs
macro_kubernetes_logsContainer logs
macro_kubernetes_stats_hostNode metrics
macro_kubernetes_stats_cgroupPod and container metrics
macro_kubernetes_proc_statsProc metrics
macro_kubernetes_net_statsNetwork metrics
macro_kubernetes_net_socket_tableNetwork socket tables
macro_kubernetes_mount_statsContainer runtime storage usage metrics
macro_kubernetes_disk_statsDisk I/O metrics
macro_kubernetes_prometheus_metricsMetrics in Prometheus format

Before 26.10, the app also defined macro_kubernetes_stats, which no dashboard read. In a search of your own, use macro_kubernetes_stats_host for node records and macro_kubernetes_stats_cgroup for pod and container records instead.

Using dedicated indexes for different types of data

For most clusters, it’s worth splitting logs from metrics: they have different access patterns, different volumes, and different retention needs. A common layout is:

  • kube_system_logs for events, container logs, and host logs;
  • kube_system_stats for proc and system metrics;
  • kube_prometheus for Prometheus metrics.

You can go further and pin every datatype Collectord forwards to its own index.

The big practical win: dedicated indexes let you set different retention policies for logs and metrics independently.

To wire it up on the Collectord side, set the index value under each input in the Configuration Reference:

yaml
  1data:
  2  002-daemonset.conf: |
  3    ...
  4
  5    [input.system_stats]
  6
  7    ...
  8
  9    index.host = kube_system_stats
 10    index.cgroup = kube_system_stats
 11
 12    ...
 13
 14    [input.proc_stats]
 15
 16    ...
 17
 18    index.host = kube_system_stats
 19    index.cgroup = kube_system_stats
 20
 21    ...
 22
 23    [input.net_stats]
 24
 25    ...
 26
 27    index.host = kube_system_stats
 28    index.cgroup = kube_system_stats
 29
 30    ...
 31
 32    [input.net_socket_table]
 33
 34    ...
 35
 36    index.host = kube_system_stats
 37    index.cgroup = kube_system_stats
 38
 39    ...
 40
 41    [input.mount_stats]
 42
 43    ...
 44
 45    index = kube_system_stats
 46
 47    ...
 48
 49    [input.disk_stats]
 50
 51    ...
 52
 53    index = kube_system_stats
 54
 55    ...
 56
 57    [input.files]
 58
 59    ...
 60
 61    index = kube_system_logs
 62
 63    ...
 64
 65    [input.app_logs]
 66
 67    ...
 68
 69    index = kube_system_logs
 70
 71    ...
 72
 73    [input.files::audit-logs]
 74
 75    ...
 76
 77    index = kube_system_logs
 78
 79    ...
 80
 81    [input.files::logs]
 82
 83    ...
 84
 85    index = kube_system_logs
 86
 87    ...
 88
 89    [input.journald]
 90
 91    ...
 92
 93    index = kube_system_logs
 94
 95    ...
 96  004-addon.conf: |
 97    ...
 98
 99    [input.kubernetes_events]
100
101    ...
102
103    index = kube_system_logs
104
105    ...
106
107    [input.kubernetes_watch::nodes]
108
109    ...
110
111    index = kube_system_logs
112
113    ...
114
115    [input.kubernetes_watch::pods]
116
117    ...
118
119    index = kube_system_logs
120
121    ...
122
123    [input.kubernetes_watch::resourcequota]
124
125    ...
126
127    index = kube_system_logs
128
129    ...

The Prometheus inputs take an index the same way, for example index = kube_prometheus:

  • each [input.prometheus::*] section in 002-daemonset.conf;
  • [input.prometheus_auto] and [input.collectord_metrics] in 001-general.conf.

[input.files::syslog] ships disabled, because the journal and [input.files::logs] carry the host logs. Give it an index only if you turn it on.

Configuring dedicated indexes, source and sourcetype for Namespaces

You can also override the destination index, source and sourcetype per namespace, workload, or pod, with annotations Collectord watches for. For example, to send every container log, metric, and event from the namespace1 namespace to the kubernetes_namespace1 index:

bash
1kubectl annotate namespaces namespace1 \
2  collectord.io/index=kubernetes_namespace1

For the full set of routing and override annotations, see Annotations.