Monitoring Docker, OpenShift, Kubernetes - Version 25.10
We are moving away from the sysver to version numbers based on the release date. As we are planning to keep backward compatibility as one of the main goals,
there is no reason to keep the version 5 forever. This version will be the first of 25.10.2, where the first two numbers represent the year and month of the release,
and the third number represents the patch version. The next major/minor version after that will be 26.04.1, with patches in between, like 25.10.3.
Now let’s talk about the new features in this release.
Store Splunk Tokens in Secrets for User Inputs
We updated the CRD to be able to reference secrets for Splunk tokens in user-defined Splunk outputs. See the updated CRD below.
1apiVersion: apiextensions.k8s.io/v1
2kind: CustomResourceDefinition
3metadata:
4 name: splunkoutputs.collectord.io
5spec:
6 group: collectord.io
7 versions:
8 - name: v1
9 served: true
10 storage: true
11 schema:
12 openAPIV3Schema:
13 type: object
14 properties:
15 spec:
16 type: object
17 properties:
18 url:
19 type: string
20 format: uri
21 insecure:
22 type: boolean
23 token:
24 type: string
25 description: "Plain token"
26 tokenFromSecret:
27 type: object
28 description: "Reference to a Kubernetes Secret"
29 properties:
30 secret:
31 type: string
32 key:
33 type: string
34 oneOf:
35 - required: ["token"]
36 - required: ["tokenFromSecret"]
37 scope: Namespaced
38 names:
39 listKind: SplunkOutputList
40 plural: splunkoutputs
41 singular: splunkoutput
42 kind: SplunkOutput
Now, if you want to create a SplunkOutput resource, you have two options: define the resource as before with the SplunkOutput resource
1apiVersion: "collectord.io/v1"
2kind: SplunkOutput
3metadata:
4 name: splunk-user-output-1
5spec:
6 url: https://splunk-hec.testing:8088/services/collector/event/1.0
7 insecure: true
8 token: 9c91ec81-856e-4293-97c3-2a8b6d4bfc0b
Or use SplunkOutput with a reference to the secret. First, you will create a secret.
1kubectl create secret generic splunk-user-output-2-secret \
2 --namespace default \
3 --from-literal=token="9c91ec81-856e-4293-97c3-2a8b6d4bfc0b"
And after that, you can define SplunkOutput
1apiVersion: "collectord.io/v1"
2kind: SplunkOutput
3metadata:
4 name: splunk-user-output-2
5spec:
6 url: https://splunk-hec.testing:8088/services/collector/event/1.0
7 insecure: true
8 tokenFromSecret:
9 secret: splunk-user-output-2-secret
10 key: token
After that you can reference those outputs with the annotations like collectord.io/output=splunk::user/default/splunk-user-output-1 or collectord.io/output=splunk::user/default/splunk-user-output-2.
You Can Configure Global Hash Patterns Within ConfigMap for Collectord
Previously, you were able to configure global replace patterns for the logs by placing pipe.replace::name configurations in ConfigMap.
Now you can configure global hash patterns for the logs by placing pipe.hash::name configurations in ConfigMap.
1# You can configure global hash rules for the events, which can help to hide sensitive data
2# from logs before they are sent to outputs. Those rules will be applied to all pipelines for container logs, host logs,
3# application logs and events.
4# In the following example we hash IP addresses with fnv-1a-64
5[pipe.hash::name]
6match = (\d{1,3}\.){3}\d{1,3}'
7function = fnv-1a-64
Filtering Objects in Inputs input.kubernetes_watch
Previously, you were able to filter events by fields attached to the Kubernetes API object that Collectord forwards to outputs. For example, you were able to define
blacklist.kubernetes_namespace: ^(default|project1|project2)$. Now Collectord can match not only by the attached fields, but also by looking inside the object’s JSON,
so you can filter by any field in the object’s JSON. For example, you can filter out all MODIFY events by adding the following configuration to ConfigMap:
1[input.kubernetes_watch::pods]
2disabled = false
3refresh = 10m
4apiVersion = v1
5kind = Pod
6type = kubernetes_objects
7excludeManagedFields = true
8blacklist.type = ^MODIFY$
Redesigned Audit (Overview) Dashboard for Kubernetes and OpenShift
We have redesigned the Audit (Overview) dashboard for Kubernetes and OpenShift. The new dashboard provides a more comprehensive view of the audit logs, including detailed information about the events, users, and resources involved. The dashboard also includes interactive filters and sorting options, making it easier to analyze and troubleshoot issues.
Helm Charts
We have had a significant number of customers who asked for Helm Charts. They are now available at github.com/outcoldsolutions/collectord-helm-charts. Please refer to the charts’ README files to learn more. With those charts, it is possible to easily install one or multiple Collectord instances, with multiple outputs defined.
Other significant changes
- We are continuing to work on improving connections to the Kubernetes API server, which included improvements on the number of connections Collectord makes on start, as well as randomization of timeouts for watch events, so they would not happen at the same time.
- You can now define a custom
sourceunderinput.files::xxx. - Collection of
prometheusmetrics from user services now includes aseedfield, which is unique for the lifetime of the input. - Added proper timestamp extraction for Audit Logs collected from the host.
Links
You can find more information about other minor updates by following the links below.
Release notes
- Monitoring OpenShift - Release notes
- Monitoring Kubernetes - Release notes
- Monitoring Docker - Release notes
