Outcold Solutions LLC

Monitoring OpenShift - Version 5

Upgrade instructions

Upgrade from version 4 to 5

Upgrade Splunk application

Download version 5.0 from SplunkBase.

Upgrade collector

  1. We mount /var/lib/docker instead of /var/lib/docker/containers to be able to search for application logs.
  2. We added new mount /var/lib/origin/openshift.local.volumes/ for not master nodes that allows to autodiscover application log in volumes created with emptyDir.
  3. We added new mount /var/lib/origin/ for master nodes that allows to autodiscover application log in volumes created with emptyDir and audit logs.
  4. We added imagePullPolicy: Always and changed visioning scheme to {major}.{minor}, where {major} can have breaking changes, and {minor} can be used with small updates. Patches for the base images will be delivered with the same version.

Download latest Configuration Reference, update you configuration with the changes from our configuration.

Update deployed configuration.

oc apply -f collectorforopenshift.yaml

Upgrade from version 5.0 to 5.1

Version 5.1 is a minor upgrade, that includes Performance improvements, Usability improvements, capability of forwarding Network Metrics, and autodiscover Prometheus metrics from Pods. For more details, please read Release History.

We include two new types of metrics, defined under stanzas input.net_stats (network metrics) and input.net_socket_table (table of network connection). The addon includes input.prometheus_auto stanza that defines auto-discovery for Prometheus metrics.

Upgrade from version 5.1 to 5.2

Version 5.2 is a minor upgrade, that includes Performance improvements, Usability improvements, and capability of forwarding Docker and Kubelet runtime storage metrics (one additional event per host once in 30 seconds). For more details, please read Release History.

Mount metrics are defined under input.mount_stats. If you override indexes for various types of data, make sure to update these metrics as well.

Additionally we introduced devnull output, that allows you to disable collection of logs or metrics for specific containers.

We moved prometheus_auto from addon to general configuration, allowing pods on host network to collect metrics from the pods running on host network, and addon to collect metrics from pods network.

With version 5.2 we predefined several alerts, that can help you to monitor the health of your clusters and performance of your applications.

Upgrade from version 5.2 to 5.3

Version 5.3 is a minor upgrade. Simple upgrade the Splunk application and the image. In the configuration file, you can find one new key group for [input.net_socket_table], that can significantly reduce licensing cost for the network socket table data.

Upgrade from version 5.3 to 5.4

Upgrade the application in Splunk and collectorforopenshift. No additional configurations has been added.

Upgrade from version 5.4 to 5.5

Upgrade the application in Splunk and collectorforopenshift. No additional configurations has been added.

Upgrade from version 5.5 to 5.6

Upgrade the application in Splunk and collectorforopenshift. There are few parts in ConfigMap are new. You have not put them in your configuration, only if you are intending to use them.

  1. Under [general.kubernetes] the keys includeAnnotations allows you to attach annotations, similarly to labels to the forwarded data. Unset by default.

  2. Under [input.files:*] two new keys samplingPercent and samplingKey for enabling sampling.

  3. The output [output.splunk] can now limit by the number of events in payload with events key.

Upgrade from version 5.6 to 5.7

Upgrade the application in Splunk and collectorforopenshift. New input is implemented input.journald, see configuration. If you have journald enabled and also forwarding messages to /var/log/messages or /var/log/syslog files, to make sure that you aren't going to forward the same host logs twice you can disable rsyslog on the system (or any other alternative) and specify from what timestamp you want Collectord to pick up journald logs

[input.journald]
startFromRel = -1h

To disable journald input

[input.journald]
disabled = true"

To disable forwarding from /var/log/messages or /var/log/syslog files use

[input.files::syslog]
disabled = true

Upgrade from version 5.7 to 5.8

Upgrade the application in Splunk and collectorforopenshift. YAML configuration file includes now critical pod annotation for OpenShift version below 3.11, and PriorityClass for OpenShift version 3.11 and above. See configuration.

Upgrade from version 5.8 to 5.9

Upgrade the application in Splunk and collectorforopenshift. See release notes for the new features (including capabilities to stream API Objects and support for multiple Splunk Clusters).

Upgrade from version 5.9 to 5.10

Upgrade the application in Splunk and collectorforopenshift. New dashboard Security/Objects(Pods) depends on streaming Pod objects from API server. See default configuration (section 004-addon.conf).

Upgrade from version 5.10 to 5.11

Upgrade the application in Splunk and collectorforkubernetes. In the YAML configuration we have added request for the persistentvolumeclaims in the ClusterRole. For several volumeMounts we added a configuration mountPropagation: HostToContainer. Update your YAML configuration to be able to use PVC for application logs.


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.