Installation

Set up Monitoring OpenShift end to end in under 10 minutes: the Splunk app and HTTP Event Collector first, then Collectord in your cluster. Collectord forwards container logs, host logs and metrics, enriched with OpenShift metadata.

If you don’t have a license key yet, request one first: see Before you begin.

Before you begin

• Splunk. Splunk Enterprise 9.2 or later, or Splunk Cloud Platform, for the Monitoring OpenShift app.
• OpenShift. 4.11 or later, the release that brought Kubernetes 1.24; the Helm chart refuses an older cluster. Nodes on amd64 or arm64; the Red Hat certified image is built for amd64 only.
• A license key. Request a 30-day evaluation key to try Collectord before production, or a free developer key for a homelab, learning or a personal project (180 days, renewable). Both forms email the key within a few minutes. For a longer evaluation, an offline key or more capacity, contact us.
• Access to registry.outcold.solutions, which the chart and the manifest pull from:
  • Chart: oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-splunk-openshift
  • Image: registry.outcold.solutions/outcoldsolutions/collectorforopenshift
  • Pulls go through ghcr.io and pkg-containers.githubusercontent.com, so allow both where egress is filtered.
  • Docker Hub has the same chart and image, with the same tags and digests: oci://registry-1.docker.io/outcoldsolutions/collectord-splunk-openshift, and docker.io/outcoldsolutions/collectorforopenshift in collectord.image.repository (or the manifest’s image: lines).
  • For the Red Hat certified image, access to registry.connect.redhat.com instead.

Set up Splunk

Install the Monitoring OpenShift app

Install the latest version of Monitoring OpenShift from Splunkbase, on your search heads only.

Choose where the stats go

Collectord always sends container, host and audit logs, and OpenShift events and objects, to an events index. The stats can go to either kind of index: the node, pod, container, process, network, filesystem and disk stats, and Prometheus metrics.

Not sure? Start with the events index. You can move later without a gap: set defaultIndexType to both, and the stats go to both indexes while you compare the two sets of dashboards.

Create the indexes

Create the indexes you chose in Settings › Indexes › New Index:

• An events index, with Index Data Type: Events, for example openshift. Optional: without one, the data goes to the HEC token’s default index.
• A metrics index, if the stats go there, with Index Data Type: Metrics, for example openshift_metrics. The default timestamp resolution works.

In a distributed deployment, create them where your data is indexed: on the indexers, through the cluster manager for an indexer cluster.

Choose the dashboards

The first time you open the app, Splunk shows its App configuration page in front of every dashboard. It stays there until an administrator (the admin or sc_admin role) chooses the dashboard set that matches where the stats go:

1. Open the app, and on the App configuration page select Continue to app setup page. Later, the same page is Setup › Configuration in the app’s menu.
2. Under Dashboards data source, choose Events-index dashboards or Metrics-index dashboards, and select Apply. Do this even when the choice already shows the set you want: until you apply it, the dashboards stay behind the App configuration page.

Both sets stay installed, and the choice only switches the app menu, so you can change it later. On a search head cluster, choose once per member. Installing the app needs no Splunk restart.

The same page holds two more settings: Alert Catalog: who sees the alerts, the roles the alerts you create from the Alert Catalog are shared with (none until you choose), and the Legacy alerts switch for the alerts earlier versions shipped.

Upgrading from an earlier version? Choose the events-index dashboards, and before you select Apply, check the Legacy alerts section: Upgrade has the steps.

Point the app at your indexes

The app finds your data through two macros:

• macro_openshift_base reads the events index. Its default, (), names no index, so it searches the indexes your role searches by default. An index you just created isn’t one of them until you add it to the role’s Indexes searched by default (Settings › Roles), so name yours.
• macro_openshift_metrics_base reads the metrics index, for the metrics-index dashboards and alerts. It is (index=*) by default, every metrics index your role can search, so it works with any name. Narrow it to yours to keep other metrics out of the app’s searches.

With the indexes from the table above, the two macros read:

1macro_openshift_base = (index=openshift)
2macro_openshift_metrics_base = (index=openshift_metrics)

To edit them, open the Macros panel on Setup › Configuration. It links each macro to its editor in Settings › Advanced search › Search macros, where editing needs the admin, power or sc_admin role. Your changes are kept across upgrades.

To split logs and stats across several events indexes, see Splunk indexes.

Create an HTTP Event Collector token

Collectord forwards data to Splunk over the HTTP Event Collector (HEC). If HEC isn’t enabled yet, follow Splunk’s guide to Set up and use HTTP Event Collector in Splunk Web.

Create a token for Collectord in Settings › Data inputs › HTTP Event Collector › New Token:

• Source type: Automatic, the wizard’s default. Collectord assigns its own source types per datatype, and the Splunk app expects them.
• Default index: your events index, if you created one.
• Allowed Indexes: the events index, plus the metrics index if the stats go there. The token refuses any other index, and Collectord then sends that data to the default index instead: the stats would land in the events index without their measurements.

If your indexes aren’t searchable by default, see Splunk Indexes for how to configure them on both the Splunk side and inside Collectord.

The rest of this guide needs two values: the HEC endpoint URL and the token. Check both with curl from a machine on your nodes’ network, a node itself for example: a check from your laptop only proves that your laptop reaches HEC. Splunk answers Success, and the test event stays in the token’s default index:

1$ curl -k https://hec.example.com:8088/services/collector/event/1.0 \
2       -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
3       -d '{"event": "hello world"}'
4{"text": "Success", "code": 0}

-k skips certificate validation; use it only for self-signed certificates.

On Splunk Cloud, HEC has a URL of its own: https://http-inputs-<stack>.splunkcloud.com:443 on a paid stack, https://<stack>.splunkcloud.com:8088 on a trial stack.

Check container log rotation

Check container log rotation before you install. The log files on each node are your buffer for any disruption between Collectord and Splunk HEC: a connectivity issue, an HEC outage, a misconfiguration.

OpenShift 4.x rotates each container’s logs across at most 5 files of 50 MiB. A container writing 50 MiB per hour then gets a 5-hour buffer before its oldest log is overwritten; at 50 MiB per minute, the buffer shrinks to 5 minutes.

To adjust rotation, follow Red Hat’s Modification of log rotation of CRI-O in Openshift 4.

If you raise the rotation, plan for the disk. With 5 files of 500 MiB on a node running about 20 containers, container logs alone take roughly 50 GiB per node.

Install Collectord

Install Collectord with the collectord-splunk-openshift Helm chart, or apply the manifest rendered from it. Either way, you set:

• Splunk HEC. The URL, the token, and certificate options if you need them.
• Your license key (see Before you begin), once you review and accept the license agreement.
• A cluster name, optionally. It lets you filter by cluster in the app when you monitor more than one.
• The metrics index, if you chose it in Choose where the stats go.

Splunk’s default HEC certificate is one no client can verify, so it needs insecure: true (insecure = true in the manifest). To verify the connection instead, see Splunk output. On Splunk Cloud:

• A trial stack’s HEC at https://<stack>.splunkcloud.com:8088 presents such a certificate and needs insecure = true.
• A paid stack’s HEC at https://http-inputs-<stack>.splunkcloud.com:443 presents a publicly trusted certificate and works with the defaults.

Log in as a user with admin rights:

1$ oc login -u kubeadmin

On a cluster that has been running for a while, Collectord starts by forwarding every historical log still on disk, which can spike both your network and your Splunk indexing. Use the [general] settings thruputPerSecond to cap throughput and tooOldEvents to skip events older than a given age. For example, tooOldEvents: 24h under collectord.configuration.general in your Helm values, or tooOldEvents = 24h under [general] in the manifest. On a Splunk trial license (500 MB a day), the first minutes can index more than a day’s allowance.

Install with Helm

You need Helm 3.8 or later. Create my_values.yaml:

 1collectord:
 2  configuration:
 3    general:
 4      acceptLicense: true
 5      license: '<your_license_key>'
 6      fields:
 7        openshift_cluster: 'development'
 8    outputs:
 9      splunk:
10        default:
11          url: 'https://hec.example.com:8088/services/collector/event/1.0'
12          token: 'B5A79AAD-D822-46CC-80D1-819F80D7BFB0'
13          insecure: true

If you chose the metrics index, set two more keys in the same file:

• defaultIndexType switches every input that sends metrics, the Prometheus inputs the control-plane dashboards read included.
• defaultMetricsIndex names the index.

1collectord:
2  configuration:
3    general:
4      defaultIndexType: metrics
5    outputs:
6      splunk:
7        default:
8          defaultMetricsIndex: openshift_metrics

Set both. Without defaultMetricsIndex, the stats go to the HEC token’s default index, an events index, and every measurement in them is lost.

For the Red Hat certified image, add repository: registry.connect.redhat.com/outcoldsolutions/collectorforopenshift under collectord.image.

Install the chart:

1$ helm install collectorforopenshift \
2    --namespace collectorforopenshift \
3    --create-namespace \
4    -f my_values.yaml \
5    oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-splunk-openshift

The chart creates the SecurityContextConstraints Collectord runs under. Its README covers the rest of its values, such as keeping the license key and the token in a Secret.

Install with the manifest

Download the latest manifest, rendered from the chart, collectorforopenshift.yaml:

1$ curl -O https://www.outcoldsolutions.com/docs/monitoring-openshift/collectorforopenshift.yaml

For the Red Hat certified image, download collectorforopenshift-rhel.yaml instead:

1$ curl -O https://www.outcoldsolutions.com/docs/monitoring-openshift/collectorforopenshift-rhel.yaml

It creates the collectorforopenshift namespace and deploys every workload Collectord needs. Open it and set the same values in its 001-general.conf. The keys sit in two sections, far apart, among many others: search the file for [general] (its first one) and for [output.splunk], and set each key below its section’s name. [license.client] comes between them with its own url and insecure; leave those empty. For example:

1[general]
2acceptLicense = true
3license = ...
4fields.openshift_cluster = development
5
6[output.splunk]
7url = https://hec.example.com:8088/services/collector/event/1.0
8token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
9insecure = true

If you chose the metrics index, also set defaultIndexType = metrics under [general] and defaultMetricsIndex = openshift_metrics under [output.splunk] of the same file.

Apply the manifest you downloaded:

1$ oc apply -f ./collectorforopenshift.yaml

For the Red Hat certified image:

1$ oc apply -f ./collectorforopenshift-rhel.yaml

Check that Collectord runs

List the workloads in Collectord’s namespace:

1$ oc get pods,daemonsets,deployments --namespace collectorforopenshift

Once the images are pulled, the pods read Running. Open the Monitoring OpenShift app in Splunk: dashboards should start populating within a minute or two. If the pods aren’t running, see Troubleshooting.

By default, Collectord forwards container logs, host logs (the journal), and metrics for hosts, pods, containers, and processes.

If you pull the Red Hat certified image from registry.connect.redhat.com and the cluster’s global pull secret doesn’t cover that registry, add a pull secret: see registry.connect.redhat.com authentication.

Uninstall

Remove Collectord with the tool you installed it with.

With Helm. Uninstall the release, and delete its namespace and its SecurityContextConstraints. The chart creates the SecurityContextConstraints ahead of the release, as a Helm hook, so helm uninstall leaves them in place:

1$ helm uninstall collectorforopenshift --namespace collectorforopenshift
2$ oc delete namespace collectorforopenshift
3$ oc delete scc collectorforopenshift

Helm never deletes a chart’s custom resource definitions. Delete them once no other Collectord release on the cluster uses them; that also deletes every Configuration and SplunkOutput object:

1$ oc delete crd configurations.collectord.io splunkoutputs.collectord.io

With the manifest. Delete everything it created, the definitions included:

1$ oc delete -f ./collectorforopenshift.yaml

For the Red Hat certified image:

1$ oc delete -f ./collectorforopenshift-rhel.yaml

Collectord keeps its state on every node, in /var/lib/collectorforopenshift/data/, named after the Helm release (collectorforopenshift above, and the manifest uses the same name). A new install on the same nodes resumes from it and doesn’t send again the logs it already sent. Its addon sends the Kubernetes events the API server still holds (on OpenShift, about the last three hours) once more, as it does at every start. To start from scratch, delete that directory on every node:

1$ oc debug node/<node> -- chroot /host rm -rf /var/lib/collectorforopenshift

In Splunk, remove what you added for Collectord once you no longer need it: the app (splunk remove app monitoringopenshift), the HEC token, and the indexes with their data.

Next steps

• Integrate the OpenShift Web Console with the app: see Integrate the OpenShift Web Console with the Monitoring OpenShift application.
• Create alerts. Pick alerts on the app’s Setup › Alert Catalog page and create the ones relevant to your environment: see Alert Catalog.
• Fix what looks off with the troubleshooting checks.
• Review audit logs. OpenShift writes the API servers’ audit logs by default, and Collectord forwards them from the control-plane nodes for the app’s Audit menu: see Audit Logs.
• Check the control plane’s metrics. The default configuration covers most clusters. If Control Plane metrics are missing in the dashboards, check that the Prometheus endpoints are reachable from Collectord: see Prometheus Metrics.
• Split your indexes. For better indexing performance, split logs and metrics into separate indexes. Splunk Indexes has the recommended layout and how to wire it up on both sides.
• Extract fields from container logs at search time: see Splunk fields extraction for container logs.
• Control forwarding per pod with annotations: custom log file paths, multi-line patterns, field extraction, index, source and sourcetype overrides, dropping noisy lines, and hashing sensitive values.