Monitoring OpenShift

Configuration

This page is the reference for the Collectord deployment manifest. It covers the two image variants we ship (the standard image on Docker Hub and the Red Hat–certified image on registry.connect.redhat.com), the OpenShift objects that get created, and the configuration patterns you’ll most often need - secrets, EC2 metadata, and dynamic indexes.

Using images on docker.io (hub.docker.com)

The standard images are built on scratch - minimal, no shell, nothing to update beyond Collectord itself. Use these unless your environment specifically requires Red Hat–certified containers.

Current most recent version of OpenShift

Looking for an older release? Manifests for previous Collectord versions live at github.com/outcoldsolutions/collectord-configurations.

Using certified images on registry.connect.redhat.com

If your environment requires Red Hat–certified containers - for compliance, support contracts, or internal policy - use the RHEL-based image instead. It’s published at outcoldsolutions/collectorforopenshift and requires a pull secret to download.

Current most recent version of OpenShift

Manifests for previous releases live at github.com/outcoldsolutions/collectord-configurations.

registry.connect.redhat.com authentication

registry.connect.redhat.com is not the same as registry.access.redhat.com. The latter hosts Red Hat’s own images and works with OpenShift out of the box; the former hosts certified partner images and requires authentication.

To pull from registry.connect.redhat.com you need to give OpenShift credentials it can use. The general procedure is documented in Allowing Pods to Reference Images from Other Secured Registries; the steps below walk through it specifically for Collectord.

After applying the manifest, switch to the namespace where Collectord runs (default collectorforopenshift):

bash
1$ oc project collectorforopenshift

On Linux, log in with Docker - the resulting ~/.docker/config.json is what we’ll convert into a secret:

bash
1$ docker login registry.connect.redhat.com
2Username: [redhat-username]
3Password: [redhat-user-password]
4Login Succeeded

Log in with your Red Hat username, not your email. Both will succeed at the login step, but only the username form lets you actually pull images.

Create the pull secret from the Docker config file:

bash
1$ oc --namespace collectorforopenshift secrets new rhcc .dockerconfigjson=$HOME/.docker/config.json

On macOS, Docker Desktop stores credentials in the Keychain instead of config.json, so the file-based approach above won’t work. Create the secret directly from the command line: oc secrets --namespace collectorforopenshift new-dockercfg rhcc --docker-server=registry.connect.redhat.com --docker-username=<user_name> --docker-password=<password> --docker-email=<email>. Make sure your password doesn’t end up in shell history - run export HISTFILE=/dev/null first, or see Execute command without keeping it in history.

Link the rhcc secret to the collectorforopenshift service account so the kubelet uses it for image pulls:

bash
1$ oc --namespace collectorforopenshift secrets link collectorforopenshift rhcc --for=pull

If pods were already created before you linked the secret, delete them so the scheduler recreates them with the right pull credentials:

bash
1oc delete --namespace collectorforopenshift pods --all

Created OpenShift Objects

Applying collectorforopenshift.yaml creates the following objects - everything is scoped to the collectorforopenshift project, and the cluster-level grants are read-only:

  • Project collectorforopenshift.
  • ClusterRole collectorforopenshift with get, list, and watch on the deployment objects Collectord needs to enrich logs and metrics with OpenShift metadata.
  • ServiceAccount collectorforopenshift - the identity Collectord uses to talk to the OpenShift API.
  • ClusterRoleBinding collectorforopenshift binds the service account to the cluster role.
  • ConfigMap collectorforopenshift carries the Collectord configuration file.
  • DaemonSet collectorforopenshift runs Collectord on non-master nodes.
  • DaemonSet collectorforopenshift-master runs Collectord on master nodes.
  • Deployment collectorforopenshift-addon is a single Collectord pod that forwards cluster-wide data (events, cluster metrics) once per cluster - not once per node.

The collectorforopenshift.yaml file itself is heavily commented; read it alongside this page for the full picture of what each input collects.

Collectord configuration

The ConfigMap collectorforopenshift holds Collectord’s configuration as ini files, with every default value visible inline.

You can override any value through environment variables using this format:

text
1COLLECTOR__{ANY_NAME}={section}__{key}={value}

Environment variables are the quickest way to try things out and debug - but for anything you’re going to keep, edit the configuration file based on the defaults shipped in collectorforopenshift.yaml. It’s easier to review and easier to diff.

Using secrets to manage configurations

When you don’t want the HEC token or license key sitting in plain text in the manifest, store them in an OpenShift secret and project them as environment variables. The example below sets up the Splunk HEC token and the license that way.

First, make sure the collectorforopenshift namespace exists - create it if it doesn’t:

bash
1oc create namespace collectorforopenshift

Create the secret with both values:

bash
1oc create secret generic collectorforopenshift \
2    --namespace collectorforopenshift \
3    --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
4    --from-literal=license="general__license="

In the YAML manifest, find the env block on each workload (two DaemonSets and one Deployment) and add:

yaml
 1env:
 2- name: COLLECTOR__SPLUNK_TOKEN
 3  valueFrom:
 4    secretKeyRef:
 5      name: collectorforopenshift
 6      key: splunk-token
 7- name: COLLECTOR__LICENSE
 8  valueFrom:
 9    secretKeyRef:
10      name: collectorforopenshift
11      key: license

Then apply the manifest as you normally would in the installation guide.

Attaching EC2 Metadata

If you’re running OpenShift on EC2 and want every event to carry instance-level context - instance ID, instance type, availability zone - point Collectord at the EC2 metadata service. Each entry maps a field name to a path from Instance Metadata and User Data:

ini
1# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
2# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
3# ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
4# ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type

For example, to attach ec2_instance_id and ec2_instance_type to every event:

ini
1[general]
2...
3ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
4ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
5...

Placeholders in indexes and sources

Static index names work for small clusters, but once you have multiple teams sharing a cluster you’ll often want logs to land in per-team or per-namespace indexes. Collectord lets you use any metadata field as a placeholder in index, source, and sourcetype.

For example, to route each namespace’s logs to its own index:

ini
1[input.files]
2
3index = oc_{{openshift_namespace}}

Or to build a structured source path that mirrors the workload hierarchy - useful when a single index holds logs from many namespaces and you want Splunk’s source-based filtering to do the heavy lifting:

ini
1[input.files]
2
3source = /{{openshift_namespace}}/{{::coalesce(openshift_daemonset_name, openshift_deployment_name, openshift_statefulset_name, openshift_cronjob_name, openshift_job_name, openshift_replicaset_name, openshift_pod_name)}}/{{openshift_pod_name}}/{{openshift_container_name}}

The coalesce function picks the first non-empty workload name, so a pod owned by a Deployment shows the Deployment name, a standalone pod shows its own name, and so on.

License server endpoint (static IPs for egress-restricted networks)

Collectord checks in with the Outcold-hosted license server on a schedule and reports telemetry to the same host. The default endpoint - license.outcold.solutions - sits behind AWS API Gateway and CloudFront, which means it resolves to a large, frequently changing range of IP addresses. That’s fine for most clusters, but it’s a problem in environments where outbound traffic has to traverse a firewall with a strict allow-list.

Available since Collectord version 26.04.2

Starting in 26.04.2, we also publish license-static.outcold.solutions, a parallel endpoint backed by AWS Global Accelerator. It serves the same license and telemetry traffic from two stable anycast IPv4 addresses, so the allow-list stays small and doesn’t drift over time:

  • 166.117.80.67
  • 99.83.183.50

The original license.outcold.solutions endpoint is unchanged - if your cluster can already reach it, there’s nothing you need to do. Only switch over if you need the static IPs.

To use the static-IP endpoint, override the two URLs under [general] in the ConfigMap (or via env var):

ini
1[general]
2licenseEndpoint = https://license-static.outcold.solutions/license/
3telemetryEndpoint = https://license-static.outcold.solutions/telemetry/

The same change as environment variables on each workload (two DaemonSets and one Deployment):

yaml
1env:
2- name: COLLECTOR__LICENSEENDPOINT
3  value: "general__licenseEndpoint=https://license-static.outcold.solutions/license/"
4- name: COLLECTOR__TELEMETRYENDPOINT
5  value: "general__telemetryEndpoint=https://license-static.outcold.solutions/telemetry/"

After that, allow-list the two IPs above for outbound tcp/443 and you’re done. The static-IP endpoint runs the same backend as the default one, so the license behaviour is identical - the only differences are the hostname and the front door.

FIPS-compliant images

Since version 26.04, we publish FIPS 140-compliant images alongside the regular images, with a -fips suffix:

outcoldsolutions/collectorforopenshift:26.04.3-fips

By default, these images use FIPS-certified cryptographic algorithms but still allow fallback to non-approved algorithms when needed. To strictly enforce FIPS 140 mode, set the environment variable GODEBUG=fips140=only - Collectord will crash if any code attempts to use a non-FIPS-140-compliant algorithm.

Collectord logs its FIPS state at startup and includes it in the output of collectord diag and collectord verify.

See FIPS 140-3 Compliance for details on the Go implementation.