Monitoring Docker, OpenShift and Kubernetes - Version 5.18

The main focus of this release was support for clusters using cgroupv2 and support for a different memory usage calculation.

Memory usage

There are a lot of ways to calculate memory usage. Since the release of the first version of Collectord, we reported the true RSS memory, which means the actual memory that a Container or the Host uses.

From the cgroupv1 definition:

Only anonymous and swap cache memory is listed as part of ‘rss’ stat. This should not be confused with the true ‘resident set size’ or the amount of physical memory used by the cgroup.
‘rss + mapped_file" will give you resident set size of cgroup.

But that might not be the same compared to what you might see in other tools, including OpenShift monitoring. They show you the memory that cannot be freed. The definition from the Metrics Server:

In an ideal world, the “working set” is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate. It includes all anonymous (non-file-backed) memory since Kubernetes does not support swap. The metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim such pages.

With release 5.18, we have changed that too, to show the memory usage in the same way as Metrics Server collects it.

Cgroupv2

You will not find a lot of clusters deployed with the cgroupv2 controller. But Docker Desktop recently changed the default cgroup controller from v1 to v2. To support enthusiasts, we added support for the cgroupv2 controller as well. Collectord automatically detects if cgroupv2 is used instead of cgroupv1 and collects the metrics in the same manner as with cgroupv1.

Allow specifying the message group name for fields extraction

Collectord allows you to extract fields from Container logs. To specify what should be a message field, you just need to keep the first unnamed group, which Collectord will automatically use as the message for the event.

We have found that to be confusing for a lot of users, and it made some fields extraction too complicated. So we have added a simple annotation collectord.io/logs-extractionMessageField that specifies which field should be used as the message.

As an example

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-extraction: '^(?P<ip_address>[^\s]+) .* \[(?P<timestamp>[^\]]+)\] (?P<message>.+)$'
    collectord.io/logs-timestampfield: timestamp
    collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
    collectord.io/logs-extractionMessageField: message
spec:
  containers:
  - name: nginx
    image: nginx

arm64 support for OpenShift

OpenShift added support for arm64 nodes, and we have implemented images with arm64 architecture as well.

Links

You can find more information about other minor updates by following the links below.

Release notes

Upgrade instructions

Installation instructions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.