Monitoring Docker, OpenShift and Kubernetes - Version 5.12 - less storage, less license, more features
Reduced size for the metrics
In previous versions, we were sending raw metrics from the proc file system and doing calculations on the Splunk side to show you correct metric values, like CPU usage and Memory usage. Now, instead, we are pre-calculating these metrics on the Collectord side and forwarding already user-friendly metric values. This provides a few improvements: less data to transfer over the network, less storage for the metrics, faster searches, as Splunk does not need to perform these evaluations anymore.
Global configurations with Custom Resources
Collectord watches for collectord.io configuration Custom Resources and applies these annotations based on the selectors.
As an example, you can apply the configuration to all Pods created with the image that contains nginx in the name as follows:
apiVersion: "collectord.io/v1"
kind: Configuration
metadata:
name: apply-to-all-nginx
annotations:
collectord.io/nginx--logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
collectord.io/nginx--logs-replace.1-val: ''
collectord.io/nginx--logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
collectord.io/nginx--logs-hashing.1-function: 'fnv-1a-64'
spec:
kubernetes_container_image: "^nginx(:.*)?$"
Watching for namespaces (projects) and workloads
Collectord watches for changes in the namespaces (OpenShift projects) and workloads. When it sees new or updated annotations, it automatically recreates pipelines for the Pods.
[general.kubernetes]
watch.namespaces = v1/namespace
watch.deployments = apps/v1/deployment
watch.configurations = apis/v1/collectord.io/configuration
Base macro for alerts
To be able to enable alerts only for Production clusters, you can change the macro_openshift_alerts_base (
macro_kubernetes_alerts_base or macro_docker_alerts_base) to macro_openshift_alerts_base = (openshift_cluster_eval=prod1 OR openshift_cluster_eval=prod2). This will generate alerts only for these two clusters.
Troubleshooting command
A new troubleshooting command has been introduced that can show you the list of all the annotations applied to a specific
container or pod. For example, you can see all the annotations applied to the specific pod postgres-pod running in namespace
default:
kubectl exec -n collectorforkubernetes collectorforkubernetes-master-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres
Beta: dynamic index names based on meta fields
You can apply dynamic index names in the configurations to forward logs or stats to a specific index based on the meta fields. For example, you can define an index as:
[input.files]
index = oc_{{openshift_namespace}}
In that case, all the logs from the namespace foo will be forwarded to the index oc_foo and all the logs from the
namespace bar will be forwarded to the index oc_bar.
This feature is in beta. We will publish additional documentation on how to set it up.
Beta: diagnostic checks
With this release, we are bringing a new capability in Collectord that will allow you to diagnose and fire alerts directly
from Collectord. For now, we implemented only one check for the Node Entropy
that can verify that the entropy_avail is above the threshold 800.
Next, we are planning to add more alerts directly to Collectord, so this data will not be forwarded to Splunk, but Collectord will generate alerts directly from the nodes. These alerts will be written as part of the standard output of the Collectord containers. For example:
ALARM-OFF "node-entropy"
ALARM-ON "node-entropy" entropy value 720 is below the threshold 800
This feature is in beta. By default, it is disabled and can be enabled in the YAML configuration for collectorforopenshift.yaml or collectorforkubernetes.yaml (under
diagnostics::node-entropy).
Links
You can find more information about other minor updates by following the links below.
Release notes
- Monitoring OpenShift - Release notes
- Monitoring Kubernetes - Release notes
- Monitoring Docker - Release notes
