Concepts

This page shows how Monitoring Kubernetes works: its two pieces, what Collectord forwards, and where a setting comes from. Read it once, and the rest of the docs make sense without piecing it together from installation, configuration and annotations.

Two pieces, one product

The product ships in two halves that share a version number:

• Collectord is the agent. It runs as a DaemonSet, control-plane nodes included, plus a single Deployment add-on. The add-on watches the Kubernetes API server for events and for objects, both the default kinds and the ones you opt into. Each Collectord pod loads the subset of the ConfigMap files relevant to its role.
• Monitoring Kubernetes is the Splunk app: dashboards, alerts, search macros and field extractions. It reads what Collectord forwarded and doesn’t talk to your cluster directly.

When you upgrade, update both to the same version. The release notes cover Splunk-app changes first, then Collectord updates:.

What Collectord forwards

Collectord covers a broad range of data sources, each configured in an [input.*] section. In Splunk, you see them in these categories:

• Logs. Container stdout / stderr, application logs from files inside the container or on a mounted volume, host logs (the journal and the *.log files under /var/log), and Kubernetes audit logs.
• Metrics. Host, pod, container and process stats from cgroups and /proc; network and socket-table metrics; mount and disk metrics; Prometheus metrics scraped from your apps; and Collectord’s own internal metrics.
• Events. Kubernetes API events.
• Objects. Kubernetes API objects, streamed through a watch. Collectord watches Pods, Nodes and ResourceQuotas by default. Opt into more kinds (Deployments, StatefulSets, ConfigMaps, Secrets, …) with [input.kubernetes_watch::*].

Each input has its own sourcetype (kubernetes_logs, kubernetes_host_logs, kubernetes_stats_v2_host, kubernetes_stats_v2_cgroup, kubernetes_proc_stats_v2, kubernetes_net_stats_v2, kubernetes_events, kubernetes_objects, kubernetes_prometheus, …) and a default index. The full list of inputs and types lives in the Configuration reference.

Inputs, outputs, and pipes

Internally, Collectord is a pipeline of three concepts:

• An input discovers data: [input.files::logs] reads /var/log, [input.kubernetes_events] watches the API server, and [input.prometheus_auto] scrapes annotated pods.
• A pipe transforms events on the way through: replace, hash, extract, override, sample, throttle.
• An output ships the result somewhere. [output.splunk] is the default, and devnull drops data. You can define several Splunk outputs and route different inputs to each.

Most annotations just configure pipes and the destination output, scoped to one pod or namespace.

Where configuration comes from

You can set the same setting in several places. They layer in this order, highest priority first:

1. Pod annotations. The most specific layer.
2. Workload annotations. Annotations on the Deployment, StatefulSet or DaemonSet propagate to its pods.
3. Namespace annotations. They apply to every pod in the namespace.
4. CRD Configuration. Cluster-level defaults, applied by selector. Platform teams use it to set policy without editing individual workloads.
5. ConfigMap files. 001-general.conf and the role-specific files (002-daemonset.conf, 004-addon.conf), mounted into the relevant Collectord pods. The lowest-priority layer.

A CRD Configuration marked with force: true overrides pod, workload and namespace annotations. It is a platform team’s escape hatch for policy that individual workloads can’t bypass: see Cluster level annotations → Forcing Cluster Level Annotations.

When the same setting is defined at several levels and the wrong value wins, run collectord describe from inside a Collectord pod: see Troubleshooting → Describe. Since version 26.04, its output tags each value with [pod], [namespace], or [configuration:<name>], so you can see exactly where it came from.

Auto-discovery vs explicit opt-in

Collectord finds some data on its own. For the rest, you point it there with an annotation or the configuration:

If something you expect isn’t in Splunk, first ask: was it auto-discovered, or do you need to point Collectord at it?

How data flows

Collectord runs on the nodes and reads from local sources (filesystem, cgroups, /proc, the API server). It pushes events to your Splunk HTTP Event Collector, and the Splunk app on the search head reads them back through macros scoped to the right indexes:

1Your container ─┐
2                ├─► [input] ──► [pipe] ──► [output] ──► Splunk HEC ──► Splunk indexer
3Your annotation ┘                                                          │
4                                                                           ▼
5Splunk role ◄── [Monitoring Kubernetes app: dashboards, alerts, macros] ◄──┘

Where to go next

• New cluster: Installation.
• Tuning what gets forwarded for a specific workload: Annotations.
• Splitting indexes by datatype, or routing to multiple Splunk clusters: Splunk indexes and Splunk HTTP Event Collector.
• Something in the dashboards looks wrong: Troubleshooting.