Monitoring Docker, OpenShift, Kubernetes - Version 5.24

^{April 28, 2025}

Version 5.24 of our applications, configurations, and Collectord are now available. In this blog post, we will cover some highlights of the release.

Forward Prometheus metrics to Splunk Metrics Index

In this release, we have added the ability to forward Prometheus metrics to the Splunk Metrics Index. We suggest configuring additional splunk output, that will point to the metrics index (or multiple metric indexes)

[output.splunk::metrics]
url = https://mysplunk.mydomain:8088/services/collector/event/1.0
token = 00000000-0000-0000-0000-000000000000

Where token is configured to write by default to the metrics index.

When configuring a Prometheus collection with annotations, you can specify to use indexType=metrics in the annotation, and optionally you can configure the index and the output.

collectord.io/prometheus.1-port: '9113'
collectord.io/prometheus.1-path: '/metrics'
collectord.io/prometheus.1-index: 'openshift_metrics'
collectord.io/prometheus.1-output: 'splunk::metrics'
collectord.io/prometheus.1-indexType: 'metrics'

After that you can use Analytics to search the metrics in the index.

Analytics

Unix time stamps can be parsed from the application logs

Now you can use format @unixtimestamp when configuring the application logs parsing.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-extraction: '^(?P<timestamp>\d+)\s$'
    collectord.io/logs-timestampfield: timestamp
    collectord.io/logs-timestampformat: '@unixtimestamp'
spec:
  containers:
  - name: nginx
    image: nginx

When you configure the application logs, you can lock files to prevent multiple readers

When you configure the application logs from PVC volumes, you can lock files to prevent multiple readers. For example, if more than one instance of the application is running, and they both use the same PVC volume, you can use the annotation

collectord.io/volume.1-logs-withlock: 'true'

And only one instance of Collectord will read the logs.

For example, in this configuration, when just one Pod is running, but annotations point to the same PVC volume, the logs will be read by only one instance of Collectord, and will be forwarded only once.

apiVersion: v1
kind: Pod
metadata:
  name: kube-load-test-volume
  annotations:
    collectord.io/volume.1-logs-name: 'logs-volume-lock'
    collectord.io/volume.1-logs-withlock: 'true'
    collectord.io/volume.1-logs-type: 'lock-test-1'
    collectord.io/volume.1-logs-onvolumedatabase: 'true'
    collectord.io/volume.2-logs-name: 'logs-volume-lock'
    collectord.io/volume.2-logs-withlock: 'true'
    collectord.io/volume.2-logs-type: 'lock-test-2'
    collectord.io/volume.2-logs-onvolumedatabase: 'true'
spec:
  restartPolicy: Never
  volumes:
    - name: logs-volume-lock
      emptyDir: {}
  containers:
    ...

In this version, we significantly improved the performance of the acknowledgment database, including concurrent usage.

Other significant changes

Included a new alert for Kubernetes and OpenShift based on the Nodes conditions: "Cluster Warning: Node Condition".
Added ability to hide process command line arguments (with annotations or globally).
Improved support for Rancher configuration, in case if volumeRootDir or container logs point to a symlink, Collectord will resolve the symlink correctly.
Various bug fixes and improvements you can find in the release notes.

Monitoring Docker, OpenShift, Kubernetes - Version 5.24

Forward Prometheus metrics to Splunk Metrics Index

Unix time stamps can be parsed from the application logs

When you configure the application logs, you can lock files to prevent multiple readers

Other significant changes

Links

Release notes

Upgrade instructions

Installation instructions

About Outcold Solutions