Upgrade - Monitoring OpenShift

If you get stuck, or want a hand planning an upgrade, contact support: we’ll help and guide you through it. The sections below cover what each release asks of an upgrade, newest first.

Always upgrade the Splunk app first and Collectord afterward. The Splunk app is backward compatible with previous Collectord versions.

In most cases, an upgrade is just a new Collectord image. To use the new features that ship with a release, you also update your configuration. Track what’s new on our blog. The configuration diffs are on GitHub at outcoldsolutions/collectord-configurations: use its tags to compare versions.

Upgrade from version 26.04 to 26.10

Version 26.10 brings together work we started in 2019 that kept waiting on other changes. Dashboards read either the events index or a Splunk metrics index, and they lead you to what needs fixing instead of only showing state. Each team creates the alerts for its own clusters and projects from the Alert Catalog.

We held each new input and field to what it adds to your license, and an upgrade keeps sending your data where it goes today.

Upgrade the application in Splunk Enterprise or Splunk Cloud first, then Collectord. Collectord now runs as one DaemonSet instead of two, so read the steps for your deployment method before you upgrade.

Upgrade the Splunk app

Upgrade the app from Splunkbase, or with Install app from file and Upgrade app checked. Splunk doesn’t need a restart. Your macros and other changes in local/ stay. Then finish the app’s setup:

Choose the dashboards. Until an administrator (the admin or sc_admin role) does this once, every dashboard shows Splunk’s “App configuration” page. Select Continue to app setup page there (later, Setup > Configuration), and choose the events-index or the metrics-index dashboards. Collectord keeps sending its stats to the events index unless you change it, so the events-index dashboards are the choice for a drop-in upgrade.
Check the legacy alerts before you finish. The Legacy alerts section should start with “Upgrading”: the alerts that shipped with earlier versions then stay as they are. The page recognizes an upgrade by this app’s alerts having run before, at least an hour before the upgrade. If it reads “New install”, finishing the setup switches the old alerts off, so switch them on there first.
Pick the new alerts. First set Alert Catalog: who sees the alerts on the Configuration page: until you do, an alert you create from the catalog is private to you, while the old ones are shared with everyone. The old alerts keep running, marked deprecated, and a later release removes them. Pick their replacements on Setup > Alert Catalog, where each template’s More names the old alert it replaces.
Retire the old alerts. In the Legacy alerts section, switch the old ones off, or choose “Keep as my own copies” to keep them past their removal.

On a search head cluster, choose the dashboards once per member, or ship local/app.conf with [install] is_configured = 1 from the deployer.

Version 26.10 can send every stats input to a Splunk metrics index too: Metrics index compares the two and shows how to move without a gap.

If you already send Prometheus metrics to a metrics index, the records now arrive packed and with dotted dimension names (metric.label.<label>, openshift.namespace). Searches written against the old dimensions need the new names: see Prometheus metrics.

What the upgrade changes in your data

More audit logs. Collectord now reads the openshift-apiserver, oauth-apiserver and oauth-server audit logs beside kube-apiserver’s, on the control-plane nodes (see Audit). At the upgrade it sends every one of those files still on disk, which can be days of them and more than a trial license allows in a day. To send only the recent ones, set tooOldEvents before you upgrade: collectord.configuration.general.tooOldEvents: 24h in your Helm values, or tooOldEvents = 24h under [general] in the manifest.

Upgrade with Helm

Move the master values. The chart no longer reads values under collectord.daemonsetMaster and collectord.configMap.daemonsetMasterAdditionalFiles. Move them under collectord.daemonset and collectord.configMap.daemonsetAdditionalFiles.
Choose the priority class. The DaemonSet now runs under the built-in system-node-critical priority class and the addon under system-cluster-critical, and the chart deletes the collectorforopenshift-critical PriorityClass. To keep the old class, set collectord.priorityClass.name: collectorforopenshift-critical.
Run helm upgrade with your values file. The chart and the image now come from registry.outcold.solutions: where egress is filtered, allow it as Installation lists, or keep pulling from Docker Hub by setting collectord.image.repository: docker.io/outcoldsolutions/collectorforopenshift. A collectord.image.repository already in your values, such as the Red Hat–certified image, stays in effect.
bash
```
1helm upgrade collectorforopenshift \
2    --namespace collectorforopenshift \
3    -f my_values.yaml \
4    oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-splunk-openshift
```
It removes the -master DaemonSet. Until its pod on a master stops, the new DaemonSet pod there may restart a few times with the data path ./data/ is locked by another collectord instance. It then continues from the same state, so file positions carry over.

If a second release reuses the first release’s class, upgrade the second release first, with its collectord.priorityClass.create and collectord.priorityClass.name values removed.

Upgrade with the manifest

Download the 26.10 manifest you installed from, collectorforopenshift.yaml or the Red Hat–certified collectorforopenshift-rhel.yaml, and set your values in it again, as Installation shows: the license, the cluster name and the HEC output in 001-general.conf.
Carry over your changes. For each key you changed in the old manifest’s 002-daemonset.conf, 003-daemonset-master.conf or 004-addon.conf, set it in the matching section of the new file; what you changed in 003-daemonset-master.conf goes into 002-daemonset.conf. Many sections have new defaults, so never copy a whole section. Add a numbered entry of yours after the section’s last number (whitelist.20 after whitelist.19), not under its old number.

Apply it:

bash

1oc apply -f ./collectorforopenshift.yaml

For the Red Hat certified image:

bash

1oc apply -f ./collectorforopenshift-rhel.yaml

Delete the old DaemonSet right away. oc apply doesn’t delete what the manifest no longer contains:
bash
```
1oc delete daemonset collectorforopenshift-master -n collectorforopenshift
```
Until you delete it, its pods keep the masters on the old configuration, and the new DaemonSet pod on each of them restarts with the data path ./data/ is locked by another collectord instance. If that pod has reached its restart back-off, delete it so it starts at once. The old pods also report FailedMount for 003-daemonset-master.conf, which the new manifest no longer has, until they are gone.
Delete the old priority class. The workloads now run under the built-in priority classes. Delete the class they ran under before, once no Collectord on the cluster uses it:
bash
```
1oc delete priorityclass collectorforopenshift-critical
```

Check the upgrade

One DaemonSet and the addon run the 26.10 image, and nothing named -master is left:

bash

1oc get daemonset,deployment -n collectorforopenshift -o wide

Then open the app: the dashboards show the data from before the upgrade and after it. Container and host logs continue where they stopped. The addon sends the Kubernetes events the API server still holds (on OpenShift, about the last three hours) once more, as it does whenever it restarts. The new dashboards check more than the old ones did, such as whether a host’s clock is synced, so a host can show a warning it didn’t show before: that is the host, not the upgrade.

Upgrade from version 25.10 to 26.04

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforopenshift image to version 26.04.X.

Liveness and readiness probes (26.04.4)

Version 26.04.4 adds /healthz (liveness) and /readyz (readiness) endpoints to Collectord’s internal HTTP server. The default collectorforopenshift.yaml now wires Kubernetes probes to them. Re-apply the updated YAML, or merge the changes by hand:

001-general.conf binds the HTTP server to 127.0.0.1:11888 and enables only httpServerEndpoints.health and httpServerEndpoints.ready. The DaemonSets run on the host network: the loopback bind keeps the endpoints off the node IP, and the kubelet still reaches them through host: 127.0.0.1 on the probe.
004-addon.conf overrides the bind to 0.0.0.0:11888, because the addon runs off the host network and the kubelet probes it on the pod IP.
Each container gains a livenessProbe (/healthz) and readinessProbe (/readyz) on port 11888.

Liveness reflects only Collectord’s internal progress, so an unreachable output never causes a restart. The metrics endpoints now default to off. If you scrape them, set httpServerEndpoints.metrics = true and widen httpServerBinding.

Upgrade from version 5.24 to 25.10

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforopenshift image to version 25.10.X.

Upgrade from version 5.23 to 5.24

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforopenshift image to version 5.24.

Upgrade from version 5.22 to 5.23

Upgrade the application in Splunk Enterprise or Splunk Cloud, and bump the collectorforopenshift image to 5.23.
To populate the Node Conditions tables in the updated app, add an [input.kubernetes_watch::nodes] section to the ConfigMap, so Collectord watches nodes.
If you run clusters with thousands of pods, set watchImplementation = 2 under [general.kubernetes] for better performance.

Upgrade from version 5.21 to 5.22

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforopenshift image to version 5.22.

Upgrade from version 5.20 to 5.21

Upgrade the application in Splunk Enterprise or Splunk Cloud, then upgrade the collectorforopenshift image to version 5.21.

Upgrade from an older version

If you run a version before 5.20, contact support: we’ll help you plan the upgrade and guide you through it.