Product news and technical notes

This week the front page of Hacker News spent a day arguing about OpenTelemetry. The trigger was Mat Duggan’s “OTel isn’t going well”, which put numbers behind a feeling many practitioners have had for a while: a handful of maintainers merging most of the work in several SDKs, semantic-convention pull requests open for 150-280 days, and language support that is first-class in Go and .NET and years behind elsewhere. The comment thread (230+ points) filled in the operator’s view: teams describing analysis-paralysis where Prometheus would have been running in an afternoon, being crushed by complexity while trying to replace a commercial APM, gateway-plus-edge collector topologies, auto-instrumentation that breaks non-trivial apps, and the recurring question of what the abstraction bought once every destination still needed its own bespoke configuration.

The defenders were there too, and they are not wrong: OTLP is a genuinely open wire format, the ecosystem is the only serious answer to vendor lock-in for traces, and more than one commenter runs it happily in production and saves five figures a month doing so.

Both camps are right, because they are answering different questions. “Is OpenTelemetry a good foundation for vendor-neutral tracing across a polyglot estate?” - yes. “Is a general-purpose telemetry pipeline the simplest way to do one specific job?” - usually not. This post is about one specific job we know well: getting Kubernetes and OpenShift container logs, metrics, and events into Splunk. That job has a purpose-built agent (Collectord, ours) and a general-purpose answer (the Splunk Distribution of the OpenTelemetry Collector for Kubernetes, Splunk’s recommended option). Below is the honest, task-by-task comparison - verified against chart version 0.158.0.

Somewhere in your cluster, a service is logging an email address, a card number, or a session token right now. Application code changes weekly, log statements are written under deadline, and code review does not reliably catch a %v that formats a whole customer struct. By the time that line is indexed in Splunk, you have a compliance problem: | delete only hides events from search, it does not remove them from the buckets on disk, and the data has already crossed the network and been replicated.

The reliable place to fix this is the node the container runs on, before the data leaves it. Collectord transforms log lines at the source: mask a value, hash it so it still correlates, drop the line, or re-route it to a restricted index - all driven by annotations that app teams put on their own workloads, and enforceable cluster-wide by the platform team.

This post walks through the whole toolbox on OpenShift with a real cluster and a real Splunk instance behind every example. Everything works identically on Kubernetes - the annotations are the same, substitute kubectl for oc and see the Kubernetes annotations docs instead of the OpenShift ones.

Splunk bills by what you ingest, and a Kubernetes cluster is an ingestion machine: every container’s stdout, kubelet probing /healthz around the clock, sidecars chattering at debug level, and application logs that repeat a timestamp Splunk already stores. Most of those bytes are never searched by anyone. They just accrue license usage and storage until the bill forces the question: what are we actually paying to keep?

The wrong answer is asking every team to log less - that campaign never ends and never sticks. The right place to cut is the collection edge, where one annotation on a workload decides what leaves the node at all. This playbook walks seven levers in the order you should pull them, each verified on a live cluster with before/after numbers from Splunk itself.

Everything below uses Collectord annotations on Kubernetes; the same annotations work identically on OpenShift (oc instead of kubectl, and openshift_* instead of kubernetes_* in field names, index names, and sourcetypes). The demo numbers were measured on an OpenShift cluster - the mechanics are the same binary on both.

Between July 28 and August 4, 2026, our license server rejected online license checks from Collectord versions 25.10.3 and older with 400 Bad Request. The root cause was a server-side change; licenses themselves remained valid throughout, and Collectord 26.04 and later were not affected. The issue is fixed, all affected agents recovered automatically, and no customer action is required. Here is the full postmortem.

There are two kinds of Kubernetes questions. What happened over the last hour? is a question for ingested data - logs and metrics already sitting in a Splunk index. What is true in my cluster right now? - is that pod still crash-looping, what do its events say, is the node Ready - is a question for the live API. Until now, answering the second kind meant leaving Splunk for a terminal and holding cluster credentials.

Kubernetes Search answers it from inside Splunk. It’s kubectl, reimagined as SPL: list resources, stream pod logs, read events, and describe objects across every cluster you register - with no agent in the cluster, no ingestion, and no index storage. It’s available on Splunkbase now, and the documentation is here.

We are happy to share the 26.04.2 release of our monitoring solutions.

Two changes in this release are likely to matter beyond the patch-level bug fixes. First, Collectord’s Prometheus endpoint/service discovery is redesigned - the underlying lookup moves from the deprecated core/v1.Endpoints API to discovery.k8s.io/v1.EndpointSlice, the config syntax becomes a set of discrete keys (endpointSlice, service, endpoint, plus port / scheme / path) instead of the legacy endpoint-http://... fake-URL string, and the ClusterRole shipped with our manifests changes accordingly. Second, we’ve stood up a parallel license-static.outcold.solutions endpoint backed by AWS Global Accelerator - two static IPv4 addresses for customers running in egress-restricted networks. The rest of the release is a handful of Splunk output bug fixes and a smaller telemetry footprint - details below.

Collectord lets app teams own how their data gets forwarded - without anyone touching a central config. The harder question is where each annotation should live: on the pod, on the workload, on the namespace, or in a cluster-level Configuration CRD. Each layer has a different audience, a different blast radius, and slightly different precedence rules.

This post is the long version of the annotations docs, aimed at platform teams running Collectord across many tenants. Examples are Kubernetes; everything carries over to OpenShift (oc instead of kubectl).

We are happy to share the 26.04 release of our monitoring solutions.

This release was heavily focused on stability and performance, with over a dozen bug fixes and a reworked log forwarding implementation for PVC volumes. On the features side, we redesigned the events investigation experience in the Splunk apps for Kubernetes and OpenShift, published FIPS 140-compliant container images for regulated environments, and brought Prometheus metrics auto-discovery to Collectord for Docker through container labels.

This blog post will guide you through the process of setting up a development environment with Docker, Kubernetes, and OpenShift, and monitoring them using Splunk. This guide is mostly referencing macOS as a development environment, but you can adjust it for other operating systems as well.

We provide configurations that work out of the box in most cases. The good thing is that most of the Kubernetes and OpenShift providers have very similar default configurations. In this blog post I will guide you through steps that you need to perform on a local development box to install all three of our main applications for Monitoring Docker, Kubernetes and OpenShift in Splunk.

We are moving away from the sysver to version numbers based on the release date. As we are planning to keep backward compatibility as one of the main goals, there is no reason to keep the version 5 forever. This version will be the first of 25.10.2, where the first two numbers represent the year and month of the release, and the third number represents the patch version. The next major/minor version after that will be 26.04.1, with patches in between, like 25.10.3.