Outcold Solutions LLC

Monitoring Docker, OpenShift and Kubernetes - Version 5.12 - less storage, less license, more features

October 22, 2019

Reduced size for the metrics

In previous versions we were sending raw metrics from the proc file system and were doing calculation on the Splunk side to show you correct metric values, like CPU usage and Memory usage. Now, instead we are pre-calculating these metrics on the Collectord side and forwarding already user-friendly metric values. That provides a few improvements: less data to transfer over the network, less storage for the metrics, faster searches, as Splunk does not need to perform these evaluations anymore.

Global configurations with Custom Resources

Collectord watches for collectord.io configuration Custom Resources, and applies these annotations based on the selectors. As an example you can apply the configuration to all Pods created with the image that contains nginx in the name as

apiVersion: "collectord.io/v1"
kind: Configuration
metadata:
  name: apply-to-all-nginx
  annotations:
    collectord.io/nginx--logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
    collectord.io/nginx--logs-replace.1-val: ''
    collectord.io/nginx--logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
    collectord.io/nginx--logs-hashing.1-function: 'fnv-1a-64'
spec:
  kubernetes_container_image: "^nginx(:.*)?$"

Watching for namespaces (projects) and workloads

Collectord watches for the changes in the namespaces (OpenShift projects) and workloads, when it sees new or updated annotations it automatically recreates pipelines for the Pods.

[general.kubernetes]
watch.namespaces = v1/namespace
watch.deployments = apis/v1/deployment
watch.configurations = apis/v1/collectord.io/configuration

Base macro for alerts

To be able to enable alerts only for the Production clusters you can change the macro_openshift_alerts_base ( macro_kubernetes_alerts_base or macro_docker_alerts_base) as macro_openshift_alerts_base = (openshift_cluster_eval=prod1 OR openshift_cluster_eval=prod2), that will generate alerts only for these two clusters.

Troubleshooting command

A new troubleshooting command has been introduces that can show you the list of all the annotations applied to a specific container or pod. For example, you can see all the annotations applied to the specific pod postgres-pod running in namespace default

kubectl exec -n collectorforkubernetes collectorforkubernetes-master-4gjmc  /collectord -- describe --namespace default --pod postgres-pod --container postgres

Beta: dynamic index names based on meta fields

You can apply dynamic index names in the configurations to forward logs or stats to a specific index, based on the meta fields. For example, you can define an index as

[input.files]

index = oc_{{openshift_namespace}}

In that case all the logs from the namespace foo will be forwarded to the index oc_foo and all the logs from the namespace bar will be forwarded to index oc_bar.

This feature is in the beta, we will publish additional documentation how to set it up.

Beta: diagnostic checks

With this release we are bringing a new capability in Collectord that will allow you to diagnose and fire alerts directly from Collectord, for now we implemented only one check for the Node Entropy that can verify that the entropy_avail is above the threshold 800.

Next we are planning to add more alerts directly to Collectord, so this data is not going to be forwarded to Splunk, but Collectord will generate alerts directly from the nodes. These alerts will be written as part of the standard output of the Collectord containers. For example

ALARM-OFF "node-entropy"
ALARM-ON "node-entropy" entropy value 720 is below the threshold 800

This feature is in the beta, by default it is disabled, and could be enabled in the YAML configuration for collectorforopenshift.yaml or collectorforkubernetes.yaml (under diagnostics::node-entropy)

You can find more information about other minor updates by following links below.

Release notes

Upgrade instructions

Installation instructions

docker, kubernetes, openshift, splunk

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.