Monitoring Kubernetes

Upgrade

The Splunk app is backward compatible with previous Collectord versions, so always upgrade the Splunk app first and Collectord afterward.

In most cases, an upgrade is just a new Collectord image — but if you want the new features that ship with a release, you’ll also need to update your configuration. Track what’s new on our blog, and see the configuration diffs on GitHub at outcoldsolutions/collectord-configurations (use tags to compare versions).

Upgrade from version 25.10 to 26.04

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforkubernetes image to version 26.04.X.

Upgrade from version 5.24 to 25.10

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforkubernetes image to version 25.10.X.

Upgrade from version 5.23 to 5.24

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforkubernetes image to version 5.24.

Upgrade from version 5.22 to 5.23

Upgrade the application in Splunk Enterprise or Splunk Cloud and bump the collectorforkubernetes image to 5.23. To populate the Node Conditions tables in the updated app, add an [input.kubernetes_watch::nodes] section to the ConfigMap so Collectord watches Nodes. If you’re running clusters with thousands of pods, set watchImplementation = 2 under [general.kubernetes] for better performance.

Upgrade from version 5.21 to 5.22

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforkubernetes image to version 5.22.

Upgrade from version 5.20 to 5.21

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforkubernetes image to version 5.21.

Upgrade from version 5.19 to 5.20

Upgrade the application in Splunk Enterprise or Splunk Cloud and bump the collectorforkubernetes image to 5.20.

If you also run containers directly on Docker — outside of Kubernetes — keep the [general.docker] section and set enableOwnWatcher = true so Collectord still watches them. If you don’t use Docker, drop the [general.docker] section from the YAML configuration entirely.

Upgrade from version 5.18 to 5.19

Upgrade the application and review the latest configuration — there are a few changes worth picking up.

Enable API Gate for Collectord

When Collectord walks pod ownership to collect metadata, it sometimes hits objects the ClusterRole doesn’t grant access to. Version 5.19 adds a way to skip those calls instead of letting them fail. Add the clusterroles resource to the collectorforkubernetes ClusterRole, then set clusterRole = collectorforkubernetes under [general.kubernetes] in the ConfigMap so Collectord knows which role to inspect.

Enable monitoring for Node Reboot Required

A new diagnostics section, [diagnostics::node-reboot-required], has been added — copy it from the configuration reference. Note that rootfs is now mounted from / to /rootfs/ instead of multiple subdirectories. If you don’t plan to enable this diagnostic, you can keep the previous mounts.

Upgrade openAPIV3Schema for CustomResourceDefinition

If you plan to use force on Cluster Level Configurations, update the openAPIV3Schema on the configurations.collectord.io CustomResourceDefinition.

Upgrade from version 5.17 to 5.18

Upgrade the Splunk application and the collectorforkubernetes image.

Upgrade from version 5.16 to 5.17

Upgrade the Splunk application and the collectorforkubernetes image. To use the new Resource Quotas dashboard, copy the [input.kubernetes_watch::resourcequota] section from the latest YAML configuration on our website into your ConfigMap so Resource Quota objects start flowing.

Upgrade from version 5.15 to 5.16

Upgrade the Splunk application and the collectorforkubernetes image. To use the new Collectord metrics dashboard, enable the input.collectord_metrics input by copying it from the latest YAML configuration on our website.

Upgrade from version 5.14 to 5.15

Upgrade the Splunk application and the collectorforkubernetes image. While you’re updating the YAML configuration, add the input.prometheus:: whitelists from our reference — they cut down the volume of metrics forwarded to Splunk.

Upgrade from version 5.12 to 5.14

Upgrade the Splunk application and the collectorforkubernetes image. If you plan to use Containerd as the runtime engine, update the YAML to mount the Containerd Unix socket.

Upgrade from version 5.11 to 5.12

Upgrade the Splunk application and the collectorforkubernetes image. Monitoring Kubernetes 5.12 is backward compatible with the previous Collectord version.

input.system_stats now has dedicated disabled, type, and output values — Collectord still accepts the unified values from previous configurations for backward compatibility. The application adds two new macros, macro_kubernetes_stats_host and macro_kubernetes_stats_cgroup, both of which depend on macro_kubernetes_stats so existing overrides keep working. Several inputs also gained new types, including input.system_stats, input.proc_stats, and input.net_stats.

In collectorforkubernetes.yaml, we added a CustomResourceDefinition for configurations.collectord.io.

Collectord can now watch namespaces and workloads automatically — update the configuration stanza:

001-general.conf ini
1[general.kubernetes]
2watch.namespaces = v1/namespace
3watch.deployments = apps/v1/deployment
4watch.configurations = collectord.io/v1/configuration

Upgrade from version 5.10 to 5.11

Upgrade the Splunk application and the collectorforkubernetes image. The YAML configuration now requests persistentvolumeclaims in the ClusterRole and adds mountPropagation: HostToContainer to several volumeMounts — pick those up so you can use PVCs for application logs.

Upgrade from version 5.9 to 5.10

Upgrade the Splunk application and the collectorforkubernetes image. The new Security/Objects(Pods) dashboard relies on streaming Pod objects from the API server — see the default configuration (section 004-addon.conf).

Upgrade from version 5.8 to 5.9

Upgrade the Splunk application and the collectorforkubernetes image. The release notes cover the new features, including streaming API Objects and support for multiple Splunk clusters.

Upgrade from version 5.7 to 5.8

Upgrade the Splunk application and the collectorforkubernetes image. The YAML configuration now ships a critical pod annotation for Kubernetes versions below 1.14 and a PriorityClass for 1.14 and above — see configuration.

Upgrade from version 5.6 to 5.7

Upgrade the Splunk application and the collectorforkubernetes image. This release adds an input.journald input — see configuration.

If journald is enabled and you’re also forwarding /var/log/messages or /var/log/syslog, you’ll end up indexing host logs twice. To avoid that, disable rsyslog (or whatever else is writing to those files) and tell Collectord where to start reading journald from:

ini
1[input.journald]
2startFromRel=-1h

To turn off the journald input entirely:

ini
1[input.journald]
2disabled=true

To stop forwarding from /var/log/messages or /var/log/syslog:

ini
1[input.files::syslog]
2disabled = true

Upgrade from version 5.5 to 5.6

Upgrade the Splunk application and the collectorforkubernetes image. The ConfigMap has a few new pieces — only add them if you plan to use them:

  1. Under [general.kubernetes], includeAnnotations attaches annotations (similar to labels) to forwarded data. Unset by default.

  2. Under [input.files:*], the new samplingPercent and samplingKey keys enable sampling.

  3. The [output.splunk] output can now cap the number of events per payload via the events key.

Upgrade from version 5.4 to 5.5

Upgrade the Splunk application and the collectorforkubernetes image. No configuration changes required.

Upgrade from version 5.3 to 5.4

Upgrade the Splunk application and the collectorforkubernetes image. No configuration changes required.

Upgrade from version 5.2 to 5.3

Version 5.3 is a minor upgrade — bump the Splunk application and the image. The configuration adds one new key, group, under [input.net_socket_table], which can significantly reduce licensing costs for network socket table data.

Upgrade from version 5.1 to 5.2

Version 5.2 is a minor upgrade with performance and usability improvements, plus forwarding of Docker and Kubelet runtime storage metrics (one extra event per host every 30 seconds). For details, see the Release History.

Mount metrics live under input.mount_stats. If you override indexes per data type, update those overrides for the new metrics too.

This release also introduces the devnull output for disabling log or metric collection on specific containers.

We moved prometheus_auto from the addon into the general configuration so pods on the host network can scrape pods on the host network, while the addon scrapes pods on the pod network.

Version 5.2 ships predefined alerts that help monitor cluster health and application performance.

Upgrade from version 5.0 to 5.1

Version 5.1 is a minor upgrade with performance and usability improvements, network metrics forwarding, and Prometheus auto-discovery from pods. For details, see the Release History.

Two new metric types are defined under input.net_stats (network metrics) and input.net_socket_table (network connection table). The addon adds an input.prometheus_auto stanza for Prometheus auto-discovery.

Upgrade from version 4 to 5

Upgrade Splunk application

Download version 5.0 from SplunkBase.

Upgrade collector

  1. We mount /var/lib/docker instead of /var/lib/docker/containers to be able to search for application logs.

  2. We added a new mount /var/lib/kubelet/ that allows auto-discovery of application logs in volumes created with emptyDir.

  3. We added imagePullPolicy: Always and changed the versioning scheme to {major}.{minor}, where {major} can have breaking changes, and {minor} can be used with small updates. Patches for the base images will be delivered with the same version.

Download the latest Configuration Reference and update your configuration with the changes from our configuration.

Update deployed configuration.

bash
1kubectl apply -f collectorforkubernetes.yaml