Product news and technical notes

This week the front page of Hacker News spent a day arguing about OpenTelemetry. The trigger was Mat Duggan’s “OTel isn’t going well”, which put numbers behind a feeling many practitioners have had for a while: a handful of maintainers merging most of the work in several SDKs, semantic-convention pull requests open for 150-280 days, and language support that is first-class in Go and .NET and years behind elsewhere. The comment thread (230+ points) filled in the operator’s view: teams describing analysis-paralysis where Prometheus would have been running in an afternoon, being crushed by complexity while trying to replace a commercial APM, gateway-plus-edge collector topologies, auto-instrumentation that breaks non-trivial apps, and the recurring question of what the abstraction bought once every destination still needed its own bespoke configuration.

The defenders were there too, and they are not wrong: OTLP is a genuinely open wire format, the ecosystem is the only serious answer to vendor lock-in for traces, and more than one commenter runs it happily in production and saves five figures a month doing so.

Both camps are right, because they are answering different questions. “Is OpenTelemetry a good foundation for vendor-neutral tracing across a polyglot estate?” - yes. “Is a general-purpose telemetry pipeline the simplest way to do one specific job?” - usually not. This post is about one specific job we know well: getting Kubernetes and OpenShift container logs, metrics, and events into Splunk. That job has a purpose-built agent (Collectord, ours) and a general-purpose answer (the Splunk Distribution of the OpenTelemetry Collector for Kubernetes, Splunk’s recommended option). Below is the honest, task-by-task comparison - verified against chart version 0.158.0.

Somewhere in your cluster, a service is logging an email address, a card number, or a session token right now. Application code changes weekly, log statements are written under deadline, and code review does not reliably catch a %v that formats a whole customer struct. By the time that line is indexed in Splunk, you have a compliance problem: | delete only hides events from search, it does not remove them from the buckets on disk, and the data has already crossed the network and been replicated.

The reliable place to fix this is the node the container runs on, before the data leaves it. Collectord transforms log lines at the source: mask a value, hash it so it still correlates, drop the line, or re-route it to a restricted index - all driven by annotations that app teams put on their own workloads, and enforceable cluster-wide by the platform team.

This post walks through the whole toolbox on OpenShift with a real cluster and a real Splunk instance behind every example. Everything works identically on Kubernetes - the annotations are the same, substitute kubectl for oc and see the Kubernetes annotations docs instead of the OpenShift ones.

Splunk bills by what you ingest, and a Kubernetes cluster is an ingestion machine: every container’s stdout, kubelet probing /healthz around the clock, sidecars chattering at debug level, and application logs that repeat a timestamp Splunk already stores. Most of those bytes are never searched by anyone. They just accrue license usage and storage until the bill forces the question: what are we actually paying to keep?

The wrong answer is asking every team to log less - that campaign never ends and never sticks. The right place to cut is the collection edge, where one annotation on a workload decides what leaves the node at all. This playbook walks seven levers in the order you should pull them, each verified on a live cluster with before/after numbers from Splunk itself.

Everything below uses Collectord annotations on Kubernetes; the same annotations work identically on OpenShift (oc instead of kubectl, and openshift_* instead of kubernetes_* in field names, index names, and sourcetypes). The demo numbers were measured on an OpenShift cluster - the mechanics are the same binary on both.

Between July 28 and August 4, 2026, our license server rejected online license checks from Collectord versions 25.10.3 and older with 400 Bad Request. The root cause was a server-side change; licenses themselves remained valid throughout, and Collectord 26.04 and later were not affected. The issue is fixed, all affected agents recovered automatically, and no customer action is required. Here is the full postmortem.

There are two kinds of Kubernetes questions. What happened over the last hour? is a question for ingested data - logs and metrics already sitting in a Splunk index. What is true in my cluster right now? - is that pod still crash-looping, what do its events say, is the node Ready - is a question for the live API. Until now, answering the second kind meant leaving Splunk for a terminal and holding cluster credentials.

Kubernetes Search answers it from inside Splunk. It’s kubectl, reimagined as SPL: list resources, stream pod logs, read events, and describe objects across every cluster you register - with no agent in the cluster, no ingestion, and no index storage. It’s available on Splunkbase now, and the documentation is here.

We are happy to share the 26.04.2 release of our monitoring solutions.

Two changes in this release are likely to matter beyond the patch-level bug fixes. First, Collectord’s Prometheus endpoint/service discovery is redesigned - the underlying lookup moves from the deprecated core/v1.Endpoints API to discovery.k8s.io/v1.EndpointSlice, the config syntax becomes a set of discrete keys (endpointSlice, service, endpoint, plus port / scheme / path) instead of the legacy endpoint-http://... fake-URL string, and the ClusterRole shipped with our manifests changes accordingly. Second, we’ve stood up a parallel license-static.outcold.solutions endpoint backed by AWS Global Accelerator - two static IPv4 addresses for customers running in egress-restricted networks. The rest of the release is a handful of Splunk output bug fixes and a smaller telemetry footprint - details below.

Collectord lets app teams own how their data gets forwarded - without anyone touching a central config. The harder question is where each annotation should live: on the pod, on the workload, on the namespace, or in a cluster-level Configuration CRD. Each layer has a different audience, a different blast radius, and slightly different precedence rules.

This post is the long version of the annotations docs, aimed at platform teams running Collectord across many tenants. Examples are Kubernetes; everything carries over to OpenShift (oc instead of kubectl).

We are happy to share the 26.04 release of our monitoring solutions.

This release was heavily focused on stability and performance, with over a dozen bug fixes and a reworked log forwarding implementation for PVC volumes. On the features side, we redesigned the events investigation experience in the Splunk apps for Kubernetes and OpenShift, published FIPS 140-compliant container images for regulated environments, and brought Prometheus metrics auto-discovery to Collectord for Docker through container labels.

This blog post will guide you through the process of setting up a development environment with Docker, Kubernetes, and OpenShift, and monitoring them using Splunk. This guide is mostly referencing macOS as a development environment, but you can adjust it for other operating systems as well.

We provide configurations that work out of the box in most cases. The good thing is that most of the Kubernetes and OpenShift providers have very similar default configurations. In this blog post I will guide you through steps that you need to perform on a local development box to install all three of our main applications for Monitoring Docker, Kubernetes and OpenShift in Splunk.

We are moving away from the sysver to version numbers based on the release date. As we are planning to keep backward compatibility as one of the main goals, there is no reason to keep the version 5 forever. This version will be the first of 25.10.2, where the first two numbers represent the year and month of the release, and the third number represents the patch version. The next major/minor version after that will be 26.04.1, with patches in between, like 25.10.3.

Version 5.24 of our applications, configurations, and Collectord is now available. In this blog post, we will cover some highlights of the release.

Version 5.23 of our applications, configurations, and Collectord are now available. In this blog post, we will cover some highlights of the release.

Version 5.22 of our applications, configurations, and Collectord is now available. In this blog post, we will cover some highlights of the release.

Version 5.21 of our applications, configurations, and Collectord is now available. In this blog post, we will cover some highlights of the release.

Version 5.20 of our applications, configurations, and Collectord is now available. In this blog post, we will cover some highlights of the release.

Large teams might have different requirements for the log management system. Some teams might prefer to use Elasticsearch or OpenSearch for log management. In this version of Collectord, we have added support for sending logs to Elasticsearch and OpenSearch.

You can install Collectord with Elasticsearch or OpenSearch support and run it in the same cluster as Collectord for Splunk. In that case, you can configure Collectord to send logs to both Splunk and Elasticsearch or OpenSearch.

Collectord version 5.20 and later supports sending logs to Elasticsearch and OpenSearch.

The main focus of this release was to implement feature requests that we received from our users and various configuration updates for the latest versions of Kubernetes, OpenShift and Docker.

The main focus of this release was support for clusters using cgroupv2 and support for a different memory usage calculation.

This release includes a lot of usability improvements for troubleshooting and configuring Collectord, and monitoring your clusters.

We have been working on an interesting case with one of our customers. Every role in Splunk has a defined disk limit, and by default the user role has only 100MB.

We are always cautious about how much data we bring to Splunk Dashboards and limit everything to make sure our applications can handle large clusters in our applications.

One search that was causing an issue was a search used to populate filters in various places of our “Monitoring OpenShift” application. Depending on the number of nodes, namespaces and labels, we expect this search to return many thousands of values, but should not take a lot of disk space.

In this blog post, we will show you how you can configure your ingesting pipeline with Splunk HTTP Event Collector to get the best performance of your Splunk Configuration. We will focus on which metrics to monitor and suggestions about when you need to Scale your Splunk deployments.

Yesterday we released a patch for Monitoring Kubernetes, OpenShift, and Docker, with version 5.16.361. This patch release focused on stability and compatibility improvements for the latest versions of Kubernetes and OpenShift.

Compared to OpenShift 3.11, OpenShift 4.x looks completely different. In the first releases of OpenShift 4.x we suggested missing features, one of them was integration with the Web Console.

We have helped many of our customers forward various logs from their Kubernetes and OpenShift environments to Splunk. We have learned a lot, and that has helped us build many features in Collectord. And we do understand that some of the features could be hard to discover, so we would like to share our guide on how to set up proper forwarding of application logs to Splunk.

We are thrilled to announce a beta version of Collectord solutions that will help you forward Kubernetes and OpenShift logs to QRadar (Syslog).

The major feature of this release is self-monitoring of Collectord. With the metrics published to Splunk from Collectord, you can easily monitor the performance of the logging pipeline and Splunk HEC input. We have included many small bug fixes and usability improvements in this release as well.

This release includes a lot of minor improvements for stability and usability and several new features. We will highlight just a few of them. For the full list, please go to the release notes (links below this page).

We have released a patch release for Collectord that includes a few new features, performance and memory improvements.

Version 5.14 of our applications and Collectord is available.

Today we are happy to announce a new addition to the family of our applications that helps you to monitor your infrastructure in Splunk Enterprise and Splunk Cloud. We have released the Monitoring Linux application together with the collectorforlinux package, built on top of the Collectord forwarder.

Version 5.12 of our applications and Collectord is available.

Minor update 5.11 focuses on stability improvements and adds support for PVC volumes for collecting application logs.

If you are using NVIDIA GPU devices for your workloads, including machine learning (ML), high performance computing (HPC), financial analytics, and video transcoding, you want to be able to monitor how efficiently you are using these devices.

Today we have shipped an updated version of Collectord (version 5.10.252) that brings two features: configuration for throughput and time correction.

Minor update 5.10 is focused on the usability improvements for the multi-cluster monitoring and security monitoring use cases.

With this release we improved capabilities for streaming data to multiple Splunk Clusters and support for deploying multiple Collectord instances on the same node (in case you need to stream the same data to multiple clusters), and added a new capability to stream objects and changes from the API Server.

With version 5.8 we include usability improvements for dashboards and various bug fixes.

If you are using Kubernetes on Mesosphere DC/OS you can find that our default configuration does not provide all the metrics and information out of the box. In this blog post we will guide you through all the configuration changes to get all the information you need to monitor the health of your clusters and performance of your applications.

Not all logs that are created are equal. Some are needed for debugging purposes, some for auditing and security, some for troubleshooting. Depending on the type of logs, different approaches could be used to reduce licensing cost. Let’s go over some of them.

Version 5.7 of our applications and Collectord includes bug fixes and a new input that allows you to forward logs directly from Journald.

Version 5.6 brings dark theme support, refreshed Logs dashboard (free-text search and more control), support for auto-refresh dashboards, and bug fixes. With Collectord 5.6 we included support for log sampling (random and hash-based), small improvements and bug fixes.

We have found from our customers that it is a very common request to let application developers see the data only from the OpenShift Projects and Kubernetes Namespaces that they are working on.

In this blog post, we will show you how you can do that with Splunk roles. We will use OpenShift as an example, but you can follow the same guidance to perform the same actions on Kubernetes namespaces.

The first release of 2019 is out. We have added support for AWS ECS services and Docker Swarm Services, and also a dashboard for reviewing quotas and limits for the Kubernetes namespaces and OpenShift projects.

Monitoring Docker application is a very generic application that can help you to get started with various orchestration tools, which could be ECS or Docker Swarm or Docker UCP. We intentionally do not add Docker Swarm or ECS-specific information in the Monitoring Docker application, as we do not want to overload this application with the orchestration tool data that you don’t use.

We are excited to share with you a minor update of our solutions for Monitoring Docker, Kubernetes and OpenShift. This update brings bug fixes and improvements.

New version 5.3 of our solution for Monitoring Docker, Kubernetes and OpenShift in Splunk comes with an updated Collectord, our container-native software for discovering, transforming and forwarding logs, and for collecting metrics.

We are happy to share with you a minor update of our solutions for Monitoring Docker, Kubernetes and OpenShift. This update brings improved capabilities for monitoring multiple clusters within one application, better observability for the state of the forwarding data, and also insights into the Splunk Usage.

We have released a patch 5.2.180 for version 5.2 of our applications for Splunk. This patch fixes an important issue in Search Head Cluster deployments: lookup with alerts causing very often replication activities on SHC. If you are currently using version 5.2 with Splunk Search Head Clustering deployments, please upgrade ASAP.

Today we are open-sourcing a Node.js application openshift-webconsole-integration. This application allows you to integrate OpenShift web console with Splunk. It embeds two links in the OpenShift Web Console. The first link gives you the ability to navigate to the Pod or Workload dashboard, where you can review the performance of the containers, review network activity and see the logs. The second link navigates you directly to search where you can start working with the logs from this specific workload or pod.

It is good to know the limits of your infrastructure. We are continually testing Collectord in our labs. Today we want to share with you the results of the tests that we have performed on AWS EC2 instances. So you can use it as a reference for planning the capacity and the cost of your deployments. We will provide you with information on how we ran the tests and how we measured the performance.

Docker UCP is the real king of orchestration, not only does it allow you to deploy workloads using docker-compose files, including docker services and docker stacks, but also it runs Kubernetes control plane and allows you to deploy Kubernetes Workloads.

With version 5.2 we are increasing observability of your clusters by providing you with information about Storage usage for mounts where you run your runtime (docker or kubelet) and helping you to react to the issues faster with pre-built alerts. We updated several control plane dashboards to help you resolve issues raised by alerts, and improved performance for the Overview dashboard, so you will be able to find workloads and pods quicker.

Are you attending Splunk .conf18 this year? Do you want to learn more about containers?

We are bringing network metrics and socket tables with the minor update Monitoring Docker, OpenShift and Kubernetes Version 5.1. This release also includes visual and usability improvements in the application, performance and stability improvements in collectord, and new configurations to dynamically discover metrics from Pods exported in Prometheus format.

We are happy to announce the 5th version of our applications for Monitoring Docker, OpenShift, and Kubernetes.

It is essential to know when the application generates a specific log message. Timestamps help us to order the events. And they help us to build a correlation between systems.

That is why logging libraries write timestamps with the log messages. When you write log messages to systems like journald or Splunk, every event has a timestamp. These applications parse the timestamp from the message or generate it from the system time. When you write logs to plain text files, you lose the information about when the application created these messages. This is why you tell loggers to identify every line with a timestamp.

We develop applications for Splunk. We develop applications for Splunk that allow you to monitor containerized applications. We develop these applications because we love containers. Of course, we use containers in our development workflow.

Today we are happy to share our development practices with you, the way we use Splunk Image and App Inspect to build our applications.

Getting logs and metrics in your Splunk cluster is just the first step to managing your logs. The next step is to build your own custom dashboard to be able to explore the data you forward to Splunk.

Congratulations to the AWS team for shipping such a great product. Based on the data provided by CNCF, more than half of all companies who run Kubernetes choose to do so on AWS. Managing the Control Plane is not the most straightforward task. EKS does that for you. The only thing that is up to you is to bootstrap worker nodes and run your applications.

At Red Hat Summit 2018, we presented our next version of the application Monitoring OpenShift in Splunk. We are happy to announce the GA of Version 4 of Monitoring OpenShift and Kubernetes. These applications are now certified by Splunk, and they are available on SplunkBase.

On this podcast, Denis Gladkikh, Head of Engineering at Outcold Solutions, introduces his company and discusses how Outcold Solutions is helping businesses monitor containerized environments in Splunk.

One problem our collectord solves for our customers is support for multi-line messages, including lines generated by rendering pretty JSON messages.

When you can avoid it, I suggest you avoid it. As an example, if you want to see pretty JSON messages for development, you can keep a configuration flag (which can be an environment variable) that changes how messages are rendered. But if you are dealing with software you did not write, read below.

We are happy to welcome you to the blog of our company, Outcold Solutions. We are a small company co-founded by two engineers. Our company focuses on building powerful Splunk applications for monitoring and log forwarding from Docker, Kubernetes, and OpenShift clusters.