Monitoring Docker, OpenShift, Kubernetes - Version 5.24
Version 5.24 of our applications, configurations, and Collectord is now available. In this blog post, we will cover some highlights of the release.
Forward Prometheus metrics to Splunk Metrics Index
In this release, we have added the ability to forward Prometheus metrics to the Splunk Metrics Index. We suggest configuring an additional Splunk output that will point to the metrics index (or multiple metric indexes)
[output.splunk::metrics]
url = https://mysplunk.mydomain:8088/services/collector/event/1.0
token = 00000000-0000-0000-0000-000000000000
The token should be configured to write by default to the metrics index.
When configuring a Prometheus collection with annotations, you can specify to use indexType=metrics in the annotation, and optionally you can configure the index and the output.
collectord.io/prometheus.1-port: '9113'
collectord.io/prometheus.1-path: '/metrics'
collectord.io/prometheus.1-index: 'openshift_metrics'
collectord.io/prometheus.1-output: 'splunk::metrics'
collectord.io/prometheus.1-indexType: 'metrics'
After that you can use Analytics to search the metrics in the index.
Unix timestamps can be parsed from application logs
Now you can use the format @unixtimestamp when configuring application log parsing.
For example:
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
annotations:
collectord.io/logs-extraction: '^(?P<timestamp>\d+)\s$'
collectord.io/logs-timestampfield: timestamp
collectord.io/logs-timestampformat: '@unixtimestamp'
spec:
containers:
- name: nginx
image: nginx
When you configure application logs, you can lock files to prevent multiple readers
When you configure application logs from PVC volumes, you can lock files to prevent multiple readers. if more than one instance of the application is running, and they both use the same PVC volume, you can use the annotation
collectord.io/volume.1-logs-withlock: 'true'
And only one instance of Collectord will read the logs.
For example, in this configuration, when just one Pod is running and annotations point to the same PVC volume, the logs will be read by only one instance of Collectord and will be forwarded only once.
apiVersion: v1
kind: Pod
metadata:
name: kube-load-test-volume
annotations:
collectord.io/volume.1-logs-name: 'logs-volume-lock'
collectord.io/volume.1-logs-withlock: 'true'
collectord.io/volume.1-logs-type: 'lock-test-1'
collectord.io/volume.1-logs-onvolumedatabase: 'true'
collectord.io/volume.2-logs-name: 'logs-volume-lock'
collectord.io/volume.2-logs-withlock: 'true'
collectord.io/volume.2-logs-type: 'lock-test-2'
collectord.io/volume.2-logs-onvolumedatabase: 'true'
spec:
restartPolicy: Never
volumes:
- name: logs-volume-lock
emptyDir: {}
containers:
...
In this version, we significantly improved the performance of the acknowledgment database, including concurrent usage.
Other significant changes
- Included a new alert for Kubernetes and OpenShift based on Node conditions: “Cluster Warning: Node Condition”.
- Added the ability to hide process command line arguments (with annotations or globally).
- Improved support for Rancher configuration; in cases where volumeRootDir or container logs point to a symlink, Collectord will resolve the symlink correctly.
- Various bug fixes and improvements can be found in the release notes.
Links
You can find more information about other minor updates by following the links below.
Release notes
- Monitoring OpenShift - Release notes
- Monitoring Kubernetes - Release notes
- Monitoring Docker - Release notes
