Monitoring Kubernetes

Configuration

Deployment file structure and Collectord settings

This page is the reference for the Collectord deployment manifest and the configuration file it ships with. Use it to see what the manifest creates in your cluster, to override settings safely, and for the patterns you’ll likely need after the install: secrets, EC2 metadata, dynamic indexes.

collectorforkubernetes.yaml

The manifest for the current release:

Looking for an older release? Manifests for previous Collectord versions live at github.com/outcoldsolutions/collectord-configurations.

Created Kubernetes Objects

Applying collectorforkubernetes.yaml creates the objects below. The namespaced ones live in the collectorforkubernetes namespace, and the cluster-level grants are read-only.

  • Namespace collectorforkubernetes.
  • CustomResourceDefinition configurations.collectord.io: the Configuration kind for cluster level annotations.
  • CustomResourceDefinition splunkoutputs.collectord.io: the SplunkOutput kind for user outputs, a namespace’s own HTTP Event Collector output.
  • ClusterRole collectorforkubernetes grants get, list, and watch on the objects Collectord needs to enrich logs and metrics with Kubernetes metadata. On Secrets it grants get alone, for the one Secret a SplunkOutput names.
  • ServiceAccount collectorforkubernetes is the identity Collectord uses to talk to the Kubernetes API.
  • ClusterRoleBinding collectorforkubernetes binds the service account to the cluster role.
  • ConfigMap collectorforkubernetes carries the Collectord configuration files.
  • DaemonSet collectorforkubernetes runs Collectord on the Linux nodes, control-plane nodes included.
  • Deployment collectorforkubernetes-addon is a single Collectord pod that forwards cluster-wide data once per cluster, not once per node: events, objects, and the Prometheus metrics of annotated pods it reaches on the pod network.
  • ResourceQuota collectorforkubernetes-critical-pods admits the two workloads’ priority classes (system-node-critical, system-cluster-critical) in the namespace, which some clusters require (GKE). It limits nothing.

The collectorforkubernetes.yaml file itself is heavily commented. Read it alongside this page for the full picture of what each input collects.

Collectord configuration

The ConfigMap collectorforkubernetes holds Collectord’s configuration as ini files, with every default value visible inline.

You can override any value with an environment variable in this format:

text
1COLLECTOR__{ANY_NAME}={section}__{key}={value}

Environment variables are the quickest way to try things out and debug. For anything you’re going to keep, edit the configuration file, starting from the defaults shipped in collectorforkubernetes.yaml: a file is easier to review and easier to diff.

To reduce log volume, run your own scripts, choose how fields are spelled or raise one subsystem’s log level, see Collectord.

Using secrets to manage configurations

To keep the HEC token and the license key out of the manifest’s plain text, store them in a Kubernetes secret and project them as environment variables. The example below sets up the Splunk HEC token and the license that way.

First, create the collectorforkubernetes namespace if it doesn’t exist yet:

bash
1kubectl create namespace collectorforkubernetes

Create the secret with both values:

bash
1kubectl create secret generic collectorforkubernetes \
2    --namespace collectorforkubernetes \
3    --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
4    --from-literal=license="general__license="

In the YAML manifest, find the env block of each workload, the DaemonSet and the Deployment, and add:

yaml
 1env:
 2- name: COLLECTOR__SPLUNK_TOKEN
 3  valueFrom:
 4    secretKeyRef:
 5      name: collectorforkubernetes
 6      key: splunk-token
 7- name: COLLECTOR__LICENSE
 8  valueFrom:
 9    secretKeyRef:
10      name: collectorforkubernetes
11      key: license

Then apply the manifest as the installation guide shows.

Host logs: the journal or the syslog files

Collectord reads each node’s own logs from two places:

  • the systemd journal ([input.journald]);
  • the *.log files under /var/log ([input.files::logs]).

It no longer reads /var/log/syslog or /var/log/messages ([input.files::syslog], off by default). Where rsyslog runs beside journald, as on the RHEL family, Amazon Linux 2 and Ubuntu, rsyslog writes those files from the journal, so reading both sent every line twice. For the same reason, [input.files::logs] leaves out the files rsyslog writes per facility on Debian and Ubuntu: auth.log, kern.log, mail.log and the like.

Before 26.10, [input.files::syslog] was on, so Collectord read /var/log/syslog and /var/log/messages beside the journal.

The journal’s entries also carry more than a syslog line does: the systemd unit, priority, facility and command name, and a timestamp with its time zone.

A node without systemd has no journal. There, collectord verify reports x input(journald): FAILED, and Collectord logs a journald input: could not locate journald path warning when it starts.

On such a node, read the syslog files instead: /var/log/syslog or /var/log/messages, and the auth.log that Debian and Ubuntu keep authentication in. Turn [input.files::syslog] on and [input.journald] off. In collectorforkubernetes.yaml, set them in 002-daemonset.conf of the ConfigMap:

002-daemonset.conf ini
1[input.files::syslog]
2disabled = false
3
4[input.journald]
5disabled = true

Or as environment variables on the DaemonSet:

yaml
1env:
2- name: COLLECTOR__SYSLOG_DISABLED
3  value: "input.files::syslog__disabled=false"
4- name: COLLECTOR__JOURNALD_DISABLED
5  value: "input.journald__disabled=true"

With the collectord-splunk-kubernetes Helm chart, set them in your values:

yaml
1collectord:
2  configuration:
3    inputs:
4      hostLogs:
5        syslog:
6          disabled: false
7      journald:
8        disabled: true

A values file that carries its own copy of hostLogs keeps what it says after an upgrade, but Helm merges the chart’s new exclude into its logs. To read each line once, take the chart’s syslog and logs values. Where journald is off, take at least the chart’s syslog match: it reads auth.log, which the logs input no longer does.

Read a host-log input on some nodes only

Available since Collectord version 26.10

Some logs exist only on some nodes. An [input.files::<name>] section with a nodeSelector, a Kubernetes label selector over the node’s labels, is read only on the nodes that match. Everywhere else, collectord verify reports it as not read on this node.

The API server audit-log input uses it to read /var/log/kube-apiserver/ on the control-plane nodes alone. The Helm chart renders it from the input’s controlPlaneOnly: true as node-role.kubernetes.io/control-plane. Where your API servers run on nodes without that label, name a label they carry, in the chart’s values or in the section:

yaml
1collectord:
2  controlPlaneNodeSelector: node-role.kubernetes.io/master
002-daemonset.conf ini
1[input.files::audit-logs]
2nodeSelector = node-role.kubernetes.io/master

Two more cases to know about:

  • A node relays other hosts’ syslog. Lines rsyslog receives over the network never reach the local journal. If rsyslog writes them to files of their own, add an [input.files::<name>] section for those files. If it mixes them into /var/log/syslog or /var/log/messages, turn [input.files::syslog] on as well, and the node’s own lines arrive twice.
  • The journal is kept only in memory. Where there is no /var/log/journal directory, journald keeps the journal under /run/log/journal. Collectord reads it there, but whatever it has not sent yet when the node reboots is gone with it. To keep it, make the journal persistent on the node: mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal && journalctl --flush.

Attaching EC2 Metadata

On EC2, every event can carry instance-level context, such as the instance ID, instance type and availability zone. To add it, point Collectord at the EC2 metadata service. Each entry maps a field name to a path from Instance Metadata and User Data:

ini
1# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
2# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
3# ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
4# ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type

For example, to attach ec2_instance_id and ec2_instance_type to every event:

ini
1[general]
2...
3ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
4ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
5...

Placeholders in indexes and sources

Static index names work for small clusters. Once several teams share a cluster, you’ll often want logs to land in per-team or per-namespace indexes. Use any metadata field as a placeholder in index, source, and sourcetype.

For example, to route each namespace’s logs to its own index:

ini
1[input.files]
2
3index = kubernetes_{{kubernetes_namespace}}

Or build a structured source path that mirrors the workload hierarchy. That helps when a single index holds logs from many namespaces and you want Splunk’s source-based filtering to do the heavy lifting:

ini
1[input.files]
2
3source = /{{kubernetes_namespace}}/{{::coalesce(kubernetes_daemonset_name, kubernetes_deployment_name, kubernetes_statefulset_name, kubernetes_cronjob_name, kubernetes_job_name, kubernetes_replicaset_name, kubernetes_pod_name)}}/{{kubernetes_pod_name}}/{{kubernetes_container_name}}

The coalesce function picks the first non-empty workload name, so a pod owned by a Deployment shows the Deployment name, a standalone pod shows its own name, and so on.

License server endpoint (static IPs for egress-restricted networks)

Collectord checks in with the Outcold-hosted license server on a schedule and reports telemetry to the same host. The default endpoint, license.outcold.solutions, sits behind AWS API Gateway and CloudFront, so it resolves to a large, frequently changing range of IP addresses. That’s fine for most clusters. It’s a problem where outbound traffic has to pass a firewall with a strict allow-list.

Available since Collectord version 26.04.2

Starting in 26.04.2, we also publish license-static.outcold.solutions, a parallel endpoint backed by AWS Global Accelerator. It serves the same license and telemetry traffic from two stable anycast IPv4 addresses, so the allow-list stays small and doesn’t drift over time:

  • 166.117.80.67
  • 99.83.183.50

The original license.outcold.solutions endpoint is unchanged. If your cluster can already reach it, there’s nothing you need to do: switch over only if you need the static IPs.

To use the static-IP endpoint, override the two URLs under [general] in 001-general.conf, or with environment variables:

ini
1[general]
2licenseEndpoint = https://license-static.outcold.solutions/license/
3telemetryEndpoint = https://license-static.outcold.solutions/telemetry/

The same change as environment variables on each workload (the DaemonSet and the Deployment):

yaml
1env:
2- name: COLLECTOR__LICENSEENDPOINT
3  value: "general__licenseEndpoint=https://license-static.outcold.solutions/license/"
4- name: COLLECTOR__TELEMETRYENDPOINT
5  value: "general__telemetryEndpoint=https://license-static.outcold.solutions/telemetry/"

Then allow-list the two IPs above for outbound tcp/443. The static-IP endpoint runs the same backend as the default one, so the license behavior is identical: only the hostname and the front door differ.

FIPS-compliant images

Since version 26.04, we publish FIPS 140-compliant images alongside the regular images, with a -fips suffix:

registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1-fips

By default, these images use FIPS-certified cryptographic algorithms, but still fall back to non-approved algorithms when needed. To enforce FIPS 140 mode strictly, set the environment variable GODEBUG=fips140=only. Collectord then crashes if any code attempts to use an algorithm that isn’t FIPS 140-compliant.

Collectord logs its FIPS state at startup and includes it in the output of collectord diag and collectord verify.

See FIPS 140-3 Compliance for details on the Go implementation.