Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Docker, OpenShift, Kubernetes - Version 5.24

Version 5.24 of our applications, configurations, and Collectord is now available. In this blog post, we will cover some highlights of the release.

Forward Prometheus metrics to Splunk Metrics Index

In this release, we have added the ability to forward Prometheus metrics to the Splunk Metrics Index. We suggest configuring an additional Splunk output that will point to the metrics index (or multiple metric indexes)

[output.splunk::metrics]
url = https://mysplunk.mydomain:8088/services/collector/event/1.0
token = 00000000-0000-0000-0000-000000000000

The token should be configured to write by default to the metrics index.

When configuring a Prometheus collection with annotations, you can specify to use indexType=metrics in the annotation, and optionally you can configure the index and the output.

collectord.io/prometheus.1-port: '9113'
collectord.io/prometheus.1-path: '/metrics'
collectord.io/prometheus.1-index: 'openshift_metrics'
collectord.io/prometheus.1-output: 'splunk::metrics'
collectord.io/prometheus.1-indexType: 'metrics'

After that you can use Analytics to search the metrics in the index.

Analytics

Unix timestamps can be parsed from application logs

Now you can use the format @unixtimestamp when configuring application log parsing.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-extraction: '^(?P<timestamp>\d+)\s$'
    collectord.io/logs-timestampfield: timestamp
    collectord.io/logs-timestampformat: '@unixtimestamp'
spec:
  containers:
  - name: nginx
    image: nginx

When you configure application logs, you can lock files to prevent multiple readers

When you configure application logs from PVC volumes, you can lock files to prevent multiple readers. if more than one instance of the application is running, and they both use the same PVC volume, you can use the annotation

collectord.io/volume.1-logs-withlock: 'true'

And only one instance of Collectord will read the logs.

For example, in this configuration, when just one Pod is running and annotations point to the same PVC volume, the logs will be read by only one instance of Collectord and will be forwarded only once.

apiVersion: v1
kind: Pod
metadata:
  name: kube-load-test-volume
  annotations:
    collectord.io/volume.1-logs-name: 'logs-volume-lock'
    collectord.io/volume.1-logs-withlock: 'true'
    collectord.io/volume.1-logs-type: 'lock-test-1'
    collectord.io/volume.1-logs-onvolumedatabase: 'true'
    collectord.io/volume.2-logs-name: 'logs-volume-lock'
    collectord.io/volume.2-logs-withlock: 'true'
    collectord.io/volume.2-logs-type: 'lock-test-2'
    collectord.io/volume.2-logs-onvolumedatabase: 'true'
spec:
  restartPolicy: Never
  volumes:
    - name: logs-volume-lock
      emptyDir: {}
  containers:
    ...

In this version, we significantly improved the performance of the acknowledgment database, including concurrent usage.

Other significant changes

  • Included a new alert for Kubernetes and OpenShift based on Node conditions: “Cluster Warning: Node Condition”.
  • Added the ability to hide process command line arguments (with annotations or globally).
  • Improved support for Rancher configuration; in cases where volumeRootDir or container logs point to a symlink, Collectord will resolve the symlink correctly.
  • Various bug fixes and improvements can be found in the release notes.

You can find more information about other minor updates by following the links below.

Release notes

Upgrade instructions

Installation instructions


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.