Configuration

Configurations

collectorforkubernetes-elasticsearch.yaml

Pick the manifest that matches your destination:

• Configuration for ElasticSearch
• Configuration for OpenSearch

To reduce log volume, run your own scripts, choose how fields are spelled or raise one subsystem’s log level, see Collectord.

Deploying on OpenShift clusters

OpenShift’s default SCCs don’t allow the host-level access Collectord needs to read container logs and host metadata, so create this SCC for its service account (on OpenShift, the Helm charts create one with the same rules).

Save the following to collectorforkubernetes-scc.yaml and apply it with oc apply -f collectorforkubernetes-scc.yaml:

 1kind: SecurityContextConstraints
 2apiVersion: security.openshift.io/v1
 3metadata:
 4  name: collectorforkubernetes
 5allowHostDirVolumePlugin: true
 6allowHostIPC: true
 7allowHostNetwork: true
 8allowHostPID: true
 9allowHostPorts: true
10allowPrivilegeEscalation: true
11allowPrivilegedContainer: true
12readOnlyRootFilesystem: false
13allowedCapabilities:
14  - '*'
15allowedUnsafeSysctls:
16  - '*'
17fsGroup:
18  type: RunAsAny
19runAsUser:
20  type: RunAsAny
21seLinuxContext:
22  type: RunAsAny
23supplementalGroups:
24  type: RunAsAny
25seccompProfiles:
26  - '*'
27users:
28  - system:serviceaccount:collectorforkubernetes:collectorforkubernetes
29volumes:
30  - '*'

Host logs: the journal or the syslog files

Collectord reads each node’s own logs from the systemd journal ([input.journald]) and from the *.log files under /var/log ([input.files::logs]). It no longer reads /var/log/syslog or /var/log/messages ([input.files::syslog], off by default): where rsyslog runs beside journald - the RHEL family, Amazon Linux 2, Ubuntu - rsyslog writes those files from the journal, so reading both sent every line twice. For the same reason [input.files::logs] leaves out the files rsyslog writes per facility on Debian and Ubuntu (auth.log, kern.log, mail.log and the like). The journal’s entries carry more than a syslog line does, too: the systemd unit, priority, facility and command name, and a timestamp with its time zone.

Before 26.10, [input.files::syslog] was on, so Collectord read /var/log/syslog and /var/log/messages beside the journal.

A node without systemd has no journal. collectord verify reports x input(journald): FAILED there, and Collectord logs a journald input: could not locate journald path warning when it starts. Read the syslog files instead (/var/log/syslog or /var/log/messages, and the auth.log Debian and Ubuntu keep authentication in) - turn [input.files::syslog] on and [input.journald] off. In collectorforkubernetes-elasticsearch.yaml, set them in 002-daemonset.conf of the ConfigMap:

1[input.files::syslog]
2disabled = false
3
4[input.journald]
5disabled = true

Or as environment variables on the DaemonSet:

1env:
2- name: COLLECTOR__SYSLOG_DISABLED
3  value: "input.files::syslog__disabled=false"
4- name: COLLECTOR__JOURNALD_DISABLED
5  value: "input.journald__disabled=true"

With the collectord-elasticsearch and collectord-opensearch Helm charts, set them in your values:

1collectord:
2  configuration:
3    inputs:
4      hostLogs:
5        syslog:
6          disabled: false
7      journald:
8        disabled: true

A values file that carries its own copy of hostLogs keeps what it says after an upgrade, but Helm merges the chart’s new exclude into its logs: take the chart’s syslog and logs values to read each line once. Where journald is off, take at least the chart’s syslog match - it reads auth.log, which the logs input no longer does.

Two more cases to know about:

• A node relays other hosts’ syslog. Lines rsyslog receives over the network never reach the local journal. If rsyslog writes them to files of their own, add an [input.files::<name>] section for those files; if it mixes them into /var/log/syslog or /var/log/messages, turn [input.files::syslog] on as well, and the node’s own lines arrive twice.
• The journal is kept only in memory. Where there is no /var/log/journal directory, journald keeps the journal under /run/log/journal. Collectord reads it there, but whatever it has not sent yet when the node reboots is gone with it. To keep it, make the journal persistent on the node: mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal && journalctl --flush.

List of Created Kubernetes Objects

Applying collectorforkubernetes-elasticsearch.yaml creates the following objects:

• Namespace collectorforkubernetes - where the ServiceAccount, ConfigMap, workloads and ResourceQuota below live.
• CustomResourceDefinition configurations.collectord.io - the cluster-level Configuration resource that applies annotations by selector (see Annotations).
• CustomResourceDefinition splunkoutputs.collectord.io - the SplunkOutput resource, shared with the Collectord for Splunk manifest; this configuration doesn’t use it.
• ClusterRole collectorforkubernetes - limited to get, list, and watch on the deployed objects. Collectord uses this read-only access to enrich logs and stats with Kubernetes metadata.
• ServiceAccount collectorforkubernetes - used to authenticate against the Kubernetes API.
• ClusterRoleBinding collectorforkubernetes - binds the service account to the cluster role.
• ConfigMap collectorforkubernetes-elasticsearch - delivers configuration files to Collectord.
• DaemonSet collectorforkubernetes-elasticsearch - runs Collectord on the Linux nodes, control-plane nodes included, so each node can read its own container and host logs.
• Deployment collectorforkubernetes-elasticsearch-addon - a single Collectord instance for cluster-wide work that should only happen once (events, watched objects).
• ResourceQuota collectorforkubernetes-critical-pods admits the two workloads’ priority classes (system-node-critical, system-cluster-critical) in the namespace, which some clusters require (GKE). It limits nothing.

The collectorforkubernetes-elasticsearch.yaml file itself is heavily commented - read through it for a deeper look at every option and the source of every log type.