This page is the reference for the Collectord deployment manifest and the configuration file it ships with. Use it to see what the manifest creates in your cluster, to override settings safely, and for the patterns you’ll likely need after the install: secrets, EC2 metadata, dynamic indexes.
collectorforkubernetes.yaml
The manifest for the current release:
Looking for an older release? Manifests for previous Collectord versions live at github.com/outcoldsolutions/collectord-configurations.
Created Kubernetes Objects
Applying collectorforkubernetes.yaml creates the objects below. The namespaced ones live in the collectorforkubernetes namespace, and the cluster-level grants are read-only.
Namespacecollectorforkubernetes.CustomResourceDefinitionconfigurations.collectord.io: theConfigurationkind for cluster level annotations.CustomResourceDefinitionsplunkoutputs.collectord.io: theSplunkOutputkind for user outputs, a namespace’s own HTTP Event Collector output.ClusterRolecollectorforkubernetesgrantsget,list, andwatchon the objects Collectord needs to enrich logs and metrics with Kubernetes metadata. On Secrets it grantsgetalone, for the one Secret aSplunkOutputnames.ServiceAccountcollectorforkubernetesis the identity Collectord uses to talk to the Kubernetes API.ClusterRoleBindingcollectorforkubernetesbinds the service account to the cluster role.ConfigMapcollectorforkubernetescarries the Collectord configuration files.DaemonSetcollectorforkubernetesruns Collectord on the Linux nodes, control-plane nodes included.Deploymentcollectorforkubernetes-addonis a single Collectord pod that forwards cluster-wide data once per cluster, not once per node: events, objects, and the Prometheus metrics of annotated pods it reaches on the pod network.ResourceQuotacollectorforkubernetes-critical-podsadmits the two workloads’ priority classes (system-node-critical,system-cluster-critical) in the namespace, which some clusters require (GKE). It limits nothing.
The collectorforkubernetes.yaml file itself is heavily commented. Read it alongside this page for the full picture of what each input collects.
Collectord configuration
The ConfigMap collectorforkubernetes holds Collectord’s configuration as ini files, with every default value visible inline.
You can override any value with an environment variable in this format:
1COLLECTOR__{ANY_NAME}={section}__{key}={value}Environment variables are the quickest way to try things out and debug. For anything you’re going to keep, edit the configuration file, starting from the defaults shipped in collectorforkubernetes.yaml: a file is easier to review and easier to diff.
To reduce log volume, run your own scripts, choose how fields are spelled or raise one subsystem’s log level, see Collectord.
Using secrets to manage configurations
To keep the HEC token and the license key out of the manifest’s plain text, store them in a Kubernetes secret and project them as environment variables. The example below sets up the Splunk HEC token and the license that way.
First, create the collectorforkubernetes namespace if it doesn’t exist yet:
1kubectl create namespace collectorforkubernetesCreate the secret with both values:
1kubectl create secret generic collectorforkubernetes \
2 --namespace collectorforkubernetes \
3 --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
4 --from-literal=license="general__license="In the YAML manifest, find the env block of each workload, the DaemonSet and the Deployment, and add:
1env:
2- name: COLLECTOR__SPLUNK_TOKEN
3 valueFrom:
4 secretKeyRef:
5 name: collectorforkubernetes
6 key: splunk-token
7- name: COLLECTOR__LICENSE
8 valueFrom:
9 secretKeyRef:
10 name: collectorforkubernetes
11 key: licenseThen apply the manifest as the installation guide shows.
Host logs: the journal or the syslog files
Collectord reads each node’s own logs from two places:
- the systemd journal (
[input.journald]); - the
*.logfiles under/var/log([input.files::logs]).
It no longer reads /var/log/syslog or /var/log/messages ([input.files::syslog], off by default). Where rsyslog runs beside journald, as on the RHEL family, Amazon Linux 2 and Ubuntu, rsyslog writes those files from the journal, so reading both sent every line twice. For the same reason, [input.files::logs] leaves out the files rsyslog writes per facility on Debian and Ubuntu: auth.log, kern.log, mail.log and the like.
Before 26.10,
[input.files::syslog]was on, so Collectord read/var/log/syslogand/var/log/messagesbeside the journal.
The journal’s entries also carry more than a syslog line does: the systemd unit, priority, facility and command name, and a timestamp with its time zone.
A node without systemd has no journal. There, collectord verify reports x input(journald): FAILED, and Collectord logs a journald input: could not locate journald path warning when it starts.
On such a node, read the syslog files instead: /var/log/syslog or /var/log/messages, and the auth.log that Debian and Ubuntu keep authentication in. Turn [input.files::syslog] on and [input.journald] off. In collectorforkubernetes.yaml, set them in 002-daemonset.conf of the ConfigMap:
1[input.files::syslog]
2disabled = false
3
4[input.journald]
5disabled = trueOr as environment variables on the DaemonSet:
1env:
2- name: COLLECTOR__SYSLOG_DISABLED
3 value: "input.files::syslog__disabled=false"
4- name: COLLECTOR__JOURNALD_DISABLED
5 value: "input.journald__disabled=true"With the collectord-splunk-kubernetes Helm chart, set them in your values:
1collectord:
2 configuration:
3 inputs:
4 hostLogs:
5 syslog:
6 disabled: false
7 journald:
8 disabled: trueA values file that carries its own copy of
hostLogskeeps what it says after an upgrade, but Helm merges the chart’s newexcludeinto itslogs. To read each line once, take the chart’ssyslogandlogsvalues. Where journald is off, take at least the chart’ssyslogmatch: it readsauth.log, which thelogsinput no longer does.
Read a host-log input on some nodes only
Available since Collectord version26.10Some logs exist only on some nodes. An [input.files::<name>] section with a nodeSelector, a Kubernetes label selector over the node’s labels, is read only on the nodes that match. Everywhere else, collectord verify reports it as not read on this node.
The API server audit-log input uses it to read /var/log/kube-apiserver/ on the control-plane nodes alone. The Helm chart renders it from the input’s controlPlaneOnly: true as node-role.kubernetes.io/control-plane. Where your API servers run on nodes without that label, name a label they carry, in the chart’s values or in the section:
1collectord:
2 controlPlaneNodeSelector: node-role.kubernetes.io/master1[input.files::audit-logs]
2nodeSelector = node-role.kubernetes.io/masterTwo more cases to know about:
- A node relays other hosts’ syslog. Lines rsyslog receives over the network never reach the local journal. If rsyslog writes them to files of their own, add an
[input.files::<name>]section for those files. If it mixes them into/var/log/syslogor/var/log/messages, turn[input.files::syslog]on as well, and the node’s own lines arrive twice. - The journal is kept only in memory. Where there is no
/var/log/journaldirectory, journald keeps the journal under/run/log/journal. Collectord reads it there, but whatever it has not sent yet when the node reboots is gone with it. To keep it, make the journal persistent on the node:mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal && journalctl --flush.
Attaching EC2 Metadata
On EC2, every event can carry instance-level context, such as the instance ID, instance type and availability zone. To add it, point Collectord at the EC2 metadata service. Each entry maps a field name to a path from Instance Metadata and User Data:
1# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
2# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
3# ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
4# ec2Metadata.ec2_instance_type = /latest/meta-data/instance-typeFor example, to attach ec2_instance_id and ec2_instance_type to every event:
1[general]
2...
3ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
4ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
5...Placeholders in indexes and sources
Static index names work for small clusters. Once several teams share a cluster, you’ll often want logs to land in per-team or per-namespace indexes. Use any metadata field as a placeholder in index, source, and sourcetype.
For example, to route each namespace’s logs to its own index:
1[input.files]
2
3index = kubernetes_{{kubernetes_namespace}}Or build a structured source path that mirrors the workload hierarchy. That helps when a single index holds logs from many namespaces and you want Splunk’s source-based filtering to do the heavy lifting:
1[input.files]
2
3source = /{{kubernetes_namespace}}/{{::coalesce(kubernetes_daemonset_name, kubernetes_deployment_name, kubernetes_statefulset_name, kubernetes_cronjob_name, kubernetes_job_name, kubernetes_replicaset_name, kubernetes_pod_name)}}/{{kubernetes_pod_name}}/{{kubernetes_container_name}}The coalesce function picks the first non-empty workload name, so a pod owned by a Deployment shows the Deployment name, a standalone pod shows its own name, and so on.
License server endpoint (static IPs for egress-restricted networks)
Collectord checks in with the Outcold-hosted license server on a schedule and reports telemetry to the same host. The default endpoint, license.outcold.solutions, sits behind AWS API Gateway and CloudFront, so it resolves to a large, frequently changing range of IP addresses. That’s fine for most clusters. It’s a problem where outbound traffic has to pass a firewall with a strict allow-list.
26.04.2Starting in 26.04.2, we also publish license-static.outcold.solutions, a parallel endpoint backed by AWS Global Accelerator. It serves the same license and telemetry traffic from two stable anycast IPv4 addresses, so the allow-list stays small and doesn’t drift over time:
166.117.80.6799.83.183.50
The original license.outcold.solutions endpoint is unchanged. If your cluster can already reach it, there’s nothing you need to do: switch over only if you need the static IPs.
To use the static-IP endpoint, override the two URLs under [general] in 001-general.conf, or with environment variables:
1[general]
2licenseEndpoint = https://license-static.outcold.solutions/license/
3telemetryEndpoint = https://license-static.outcold.solutions/telemetry/The same change as environment variables on each workload (the DaemonSet and the Deployment):
1env:
2- name: COLLECTOR__LICENSEENDPOINT
3 value: "general__licenseEndpoint=https://license-static.outcold.solutions/license/"
4- name: COLLECTOR__TELEMETRYENDPOINT
5 value: "general__telemetryEndpoint=https://license-static.outcold.solutions/telemetry/"Then allow-list the two IPs above for outbound tcp/443. The static-IP endpoint runs the same backend as the default one, so the license behavior is identical: only the hostname and the front door differ.
FIPS-compliant images
Since version 26.04, we publish FIPS 140-compliant images alongside the regular images, with a -fips suffix:
registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1-fipsBy default, these images use FIPS-certified cryptographic algorithms, but still fall back to non-approved algorithms when needed. To enforce FIPS 140 mode strictly, set the environment variable GODEBUG=fips140=only. Collectord then crashes if any code attempts to use an algorithm that isn’t FIPS 140-compliant.
Collectord logs its FIPS state at startup and includes it in the output of collectord diag and collectord verify.
See FIPS 140-3 Compliance for details on the Go implementation.