Monitoring Docker, OpenShift, Kubernetes - Version 25.10
We are moving away from the sysver to version numbers based on the release date. As we are planning to keep backward compatibility as one of the main goals,
there is no reason to keep the version 5 forever. This version will be the first of 25.10.2, where the first two numbers represent the year and month of the release,
and the third number represents the patch version. The next major/minor version after that will be 26.04.1, with patches in between, like 25.10.3.
Now let’s talk about the new features in this release.
Store Splunk Tokens in Secrets for User Inputs
We updated the CRD to be able to reference secrets for Splunk tokens in user-defined Splunk outputs. See the updated CRD below.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: splunkoutputs.collectord.io
spec:
group: collectord.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
url:
type: string
format: uri
insecure:
type: boolean
token:
type: string
description: "Plain token"
tokenFromSecret:
type: object
description: "Reference to a Kubernetes Secret"
properties:
secret:
type: string
key:
type: string
oneOf:
- required: ["token"]
- required: ["tokenFromSecret"]
scope: Namespaced
names:
listKind: SplunkOutputList
plural: splunkoutputs
singular: splunkoutput
kind: SplunkOutput
Now, if you want to create a SplunkOutput resource, you have two options: define the resource as before with the SplunkOutput resource
apiVersion: "collectord.io/v1"
kind: SplunkOutput
metadata:
name: splunk-user-output-1
spec:
url: https://splunk-hec.testing:8088/services/collector/event/1.0
insecure: true
token: 9c91ec81-856e-4293-97c3-2a8b6d4bfc0b
Or use SplunkOutput with a reference to the secret. First, you will create a secret.
kubectl create secret generic splunk-user-output-2-secret \
--namespace default \
--from-literal=token="9c91ec81-856e-4293-97c3-2a8b6d4bfc0b"
And after that, you can define SplunkOutput
apiVersion: "collectord.io/v1"
kind: SplunkOutput
metadata:
name: splunk-user-output-2
spec:
url: https://splunk-hec.testing:8088/services/collector/event/1.0
insecure: true
tokenFromSecret:
secret: splunk-user-output-2-secret
key: token
After that you can reference those outputs with the annotations like collectord.io/output=splunk::user/default/splunk-user-output-1 or collectord.io/output=splunk::user/default/splunk-user-output-2.
You Can Configure Global Hash Patterns Within ConfigMap for Collectord
Previously, you were able to configure global replace patterns for the logs by placing pipe.replace::name configurations in ConfigMap.
Now you can configure global hash patterns for the logs by placing pipe.hash::name configurations in ConfigMap.
# You can configure global hash rules for the events, which can help to hide sensitive data
# from logs before they are sent to outputs. Those rules will be applied to all pipelines for container logs, host logs,
# application logs and events.
# In the following example we hash IP addresses with fnv-1a-64
[pipe.hash::name]
match = (\d{1,3}\.){3}\d{1,3}'
function = fnv-1a-64
Filtering Objects in Inputs input.kubernetes_watch
Previously, you were able to filter events by fields attached to the Kubernetes API object that Collectord forwards to outputs. For example, you were able to define
blacklist.kubernetes_namespace: ^(default|project1|project2)$. Now Collectord can match not only by the attached fields, but also by looking inside the object’s JSON,
so you can filter by any field in the object’s JSON. For example, you can filter out all MODIFY events by adding the following configuration to ConfigMap:
[input.kubernetes_watch::pods]
disabled = false
refresh = 10m
apiVersion = v1
kind = Pod
type = kubernetes_objects
excludeManagedFields = true
blacklist.type = ^MODIFY$
Redesigned Audit (Overview) Dashboard for Kubernetes and OpenShift
We have redesigned the Audit (Overview) dashboard for Kubernetes and OpenShift. The new dashboard provides a more comprehensive view of the audit logs, including detailed information about the events, users, and resources involved. The dashboard also includes interactive filters and sorting options, making it easier to analyze and troubleshoot issues.
Helm Charts
We have had a significant number of customers who asked for Helm Charts. They are now available at github.com/outcoldsolutions/collectord-helm-charts. Please refer to the charts’ README files to learn more. With those charts, it is possible to easily install one or multiple Collectord instances, with multiple outputs defined.
Other significant changes
- We are continuing to work on improving connections to the Kubernetes API server, which included improvements on the number of connections Collectord makes on start, as well as randomization of timeouts for watch events, so they would not happen at the same time.
- You can now define a custom
sourceunderinput.files::xxx. - Collection of
prometheusmetrics from user services now includes aseedfield, which is unique for the lifetime of the input. - Added proper timestamp extraction for Audit Logs collected from the host.
Links
You can find more information about other minor updates by following the links below.
Release notes
- Monitoring OpenShift - Release notes
- Monitoring Kubernetes - Release notes
- Monitoring Docker - Release notes
