This week the front page of Hacker News spent a day arguing about OpenTelemetry. The trigger was Mat Duggan’s “OTel isn’t going well”, which put numbers behind a feeling many practitioners have had for a while: a handful of maintainers merging most of the work in several SDKs, semantic-convention pull requests open for 150-280 days, and language support that is first-class in Go and .NET and years behind elsewhere. The comment thread (230+ points) filled in the operator’s view: teams describing analysis-paralysis where Prometheus would have been running in an afternoon, being crushed by complexity while trying to replace a commercial APM, gateway-plus-edge collector topologies, auto-instrumentation that breaks non-trivial apps, and the recurring question of what the abstraction bought once every destination still needed its own bespoke configuration.

The defenders were there too, and they are not wrong: OTLP is a genuinely open wire format, the ecosystem is the only serious answer to vendor lock-in for traces, and more than one commenter runs it happily in production and saves five figures a month doing so.

Both camps are right, because they are answering different questions. “Is OpenTelemetry a good foundation for vendor-neutral tracing across a polyglot estate?” - yes. “Is a general-purpose telemetry pipeline the simplest way to do one specific job?” - usually not. This post is about one specific job we know well: getting Kubernetes and OpenShift container logs, metrics, and events into Splunk. That job has a purpose-built agent (Collectord, ours) and a general-purpose answer (the Splunk Distribution of the OpenTelemetry Collector for Kubernetes, Splunk’s recommended option). Below is the honest, task-by-task comparison - verified against chart version 0.158.0.

Where OpenTelemetry is the right answer

Before the comparison, the cases where you should stop reading and use the OTel Collector:

Distributed tracing. Collectord does not collect traces. If tracing matters to you, you will run an OTel pipeline for spans regardless of what carries your logs - and the two coexist cleanly. This is our standing recommendation, not a concession.
Arbitrary OTLP backends. If the same telemetry must land in Splunk and another OTLP-speaking vendor - Datadog, Honeycomb, Grafana Cloud - the collector’s fan-out is exactly what it is for. Collectord is not single-destination (it forwards to Elasticsearch and OpenSearch, to QRadar and other SIEMs via syslog, and to several Splunk deployments at once), but its output list is those destinations, not any OTLP consumer.
An all-open-source mandate. Collectord is commercial software. If your constraint is “no license line items,” the OTel chart plus your own dashboards is the path, and the cost moves from the license column to the engineering column.

If none of those apply and the job is “cluster telemetry into Splunk,” keep reading.

The two stacks, side by side

Both deployments look similar from orbit. The Splunk OTel chart installs an agent DaemonSet (filelog receiver reading /var/log/pods, OTLP receiver) and a cluster receiver Deployment (events, objects, cluster metrics); a gateway tier is available but off by default. Collectord installs two DaemonSets (workers, control plane) and one addon Deployment (events, watched objects). Default resource limits are in the same ballpark. Both speak to Splunk HTTP Event Collector. Both are Go binaries reading the same files.

The difference is the shape of the configuration surface and who gets to touch it:

The chart’s values.yaml is ~1,600 lines exposing 458 documented properties, and the pipeline itself - receivers, processors, exporters per signal - is assembled from those values. Splunk’s own docs note that list-valued config sections are fully redefined when overridden: to add one processor to the logs pipeline, you restate the entire default processor chain in your values file and run helm upgrade.
Collectord’s pipeline ships assembled. Cluster-wide behavior lives in one ConfigMap; everything per-workload is a Kubernetes annotation on the pod, workload, or namespace - applied without redeploying or restarting the agent, and app teams can do it themselves inside their own RBAC.

That one architectural difference drives almost every row below.

Six everyday tasks, both ways

These are the tasks every Splunk-bound cluster actually performs, taken from what our customers do and what the cost and PII posts demonstrate end to end.

1. Route a namespace to its own index

Credit where due: this is the one task the Splunk OTel chart also solves with annotations, out of the box:

bash

1# Splunk OTel Collector chart
2kubectl annotate namespace payments splunk.com/index=k8s_payments
3
4# Collectord
5kubectl annotate namespace payments collectord.io/index=kubernetes_payments

The chart supports splunk.com/index, splunk.com/metricsIndex, splunk.com/sourcetype, and splunk.com/exclude at the pod and namespace level. If index routing and pod exclusion are all the self-service you need, both tools deliver it. The gaps appear the moment the task involves the content of the log stream - and note one asymmetry: the chart’s cluster receiver honors only namespace-level index annotations, so per-pod routing does not apply to events and objects.

2. Drop the lines nobody reads

Kubelet probe hits, DEBUG output, framework banners - the volume you pay for and never search.

With the OTel chart, per-line drops are a stanza filter operator added to logsCollection.containers.extraOperators in Helm values, with an expression like body matches "GET /healthz" - a platform-team change, shipped with helm upgrade, applied to every workload the expression matches. It works, with a caveat worth knowing: when chart 0.152 rewrote container-log parsing, the parsed line moved from attributes.log to body, and filter expressions referencing the old field didn’t fail - they silently evaluated to null and stopped matching.

With Collectord, the team that owns the noisy workload drops its own noise:

yaml

1collectord.io/logs-replace.1-search: '^.+"GET /(healthz|readyz)[^"]*" 200 .+$'
2collectord.io/logs-replace.1-val: ''

One annotation on one Deployment - applied with that workload’s own rollout, no cluster-wide filter expression to review, and the collector itself never restarts. (Annotation edits on already-running pods take effect within Collectord’s metadata refresh, ten minutes by default; a rollout applies them immediately.)

3. Mask sensitive data before it leaves the node

The OTel answer is the transform processor with OTTL statements - replace_pattern(body, "\\d{3}-\\d{2}-\\d{4}", "XXX-XX-XXXX") - added to the agent config through Helm values, which (per the redefinition rule above) means restating the whole default logs-processor chain to insert it. It is capable machinery: OTTL can rewrite anything. It is also a new language for the platform team to own, applied cluster-wide by people who don’t know which services log which secrets.

Collectord treats masking as a per-workload annotation (logs-replace, logs-hashing with 17 hash functions for correlation-preserving pseudonymization), with a cluster-wide [pipe.replace::]/[pipe.hash::] backstop and a Configuration CRD with force: true when the platform team must win. The PII masking post walks all of it, verified against a live cluster.

4. Sample the firehose without losing investigations

The OTel Collector’s probabilistic_sampler processor supports logs - at alpha stability, and the chart’s own sampling example covers only traces. Credit where due here as well: it samples by key natively - set attribute_source: record and from_attribute, and the same attribute value hashes to the same keep-or-drop decision, which is the property that keeps sampled data investigable. The catch is the word attribute: container log lines arrive as an unstructured body, so lifting the user or session key out of the text into a record attribute is a parser or OTTL step you build first, per log format.

Collectord ships both as annotations - logs-sampling-percent for random, logs-sampling-key where the key extraction is the annotation’s own regex - demonstrated with measured numbers in the cost playbook.

5. Cap a runaway container

A crash-looping pod stack-tracing at full speed can push gigabytes into your license overnight. In the OTel Collector there is no per-container rate limit - not in the chart, not upstream; the requests are open issues (#18476, #35204), and the available blunt instrument is the memory limiter. Collectord has collectord.io/logs-ThruputPerSecond: 128Kb per container, plus logs-TooOldEvents to keep a re-enabled workload from replaying its history.

6. Collect the log file the app writes to a volume - without a sidecar

Not everything reaches stdout. Audit logs, GC logs, slow-query logs - the files an application writes to an emptyDir or PVC because they must survive a restart. This has been Collectord’s founding use case since 2017, and it is where the two models diverge the most.

The OTel chart’s default pipeline reads container stdout from /var/log/pods and nothing else. Its one file feature, logsCollection.extraFileLogs, tails node paths you mount into the agent DaemonSet yourself. Tailing an emptyDir this way is possible - a wildcard over the pod-UID segment of the kubelet’s volume paths matches the files. What is missing is everything around the tailing: the pipeline behind extraFileLogs carries no k8sattributes processor, so nothing maps that UID back to a pod, namespace, or labels - the events arrive as anonymous node files - and the glob is cluster-global, matching every pod’s volume of that name rather than the workloads that opted in. The user in #1385 built exactly this setup, asked how to attach Kubernetes metadata, and the issue was closed with a pointer to the support portal; #984, asking for a simpler file-collection story, was closed as not-planned with the maintainer suggesting a sidecar built from the chart’s ConfigMaps. The Kubernetes-docs fallback remains the streaming sidecar: one more container per pod, re-printing files to stdout, doubling the log I/O.

With Collectord, it is - once again - an annotation on the pod:

yaml

1collectord.io/volume.1-logs-name: 'audit-logs'
2collectord.io/volume.1-logs-glob: '*.log'

Collectord discovers the files on the named volume (emptyDir, hostPath, or PVC), attaches the full pod metadata, and runs them through the same pipeline as everything else - masking, hashing, whitelist, extraction, sampling, index routing, and throughput caps all have volume.{N}-logs- variants. No sidecar, no per-pod containers to maintain, no separate pipeline. The application logs docs cover the details, including position tracking that survives pod rescheduling on PVC-backed volumes.

Day two: dashboards, upgrades, stability

What renders the data. Getting logs into an index is the start of the job, not the end. Splunk ships no dashboards for the data its OTel chart collects into Splunk Enterprise or Splunk Cloud - the Splunkbase listing is the collector itself, and the only dashboard app for those sourcetypes is a third-party, developer-supported project. Collectord’s Monitoring Kubernetes and Monitoring OpenShift apps ship 50+ dashboards and 39+ alerts built for exactly the data the agent forwards - CrashLoopBackOff triage, etcd health, capacity, events timelines, audit. That is the difference between “data arrives” and “on-call can answer questions” - and it is months of SPL engineering if you build it yourself.

Upgrade churn. The chart tracks the collector’s 0.x releases, and its own upgrade guide documents breaking changes landing in minor versions: the container-log parsing rewrite that moved fields (0.152), component renames like k8sobjects to k8s_objects that hard-fail installs still referencing old names, subcharts deprecated and removed within three releases. None of this is negligence - it is what a fast-moving 0.x ecosystem looks like - but you are subscribing to it with every helm upgrade. Collectord has shipped against the same Splunk HEC contract since 2017, on a deliberate cadence: two feature releases a year (the YY.MM versions), patch releases in between, and a release history where breaking changes are the exception rather than the release notes’ main section.

Stability labels. By the OTel project’s own component grading, the chart’s default log path runs on filelog (beta), k8sattributes (beta), splunk_hec exporter (beta) - and the filter processor that implements splunk.com/exclude is alpha, as is the Kubernetes events receiver. Beta components are broadly production-worthy and the chart is production-tested at real customers; but if your compliance framework asks “is every component in the log path GA,” the honest answer today is no.

Who fixes your bug. The Splunk distribution is supported - Splunk’s README says customers “can receive direct help from official Splunk support within SLA’s”, with best-effort guidance only for upstream OSS components - so the difference is not supported-versus-unsupported. It is what a fix looks like when it arrives. The chart has no LTS or release branches: patch releases land only on the newest 0.x before the next one ships, so taking a fix means taking the current release train with whatever else changed - which is why the chart’s own README advises pinning exact versions and reading the release notes before every upgrade. And the gaps stay gaps: both volume-log requests above were closed rather than built. Collectord’s support is the vendor that writes the code: we investigate the issue directly, and when the fix is code, a patch release ships against the version you are already running, typically within days - the 26.04 line alone carried four patch releases (26.04.1 through 26.04.4) without forcing anyone onto a new feature train.

What the HN critique does not apply to

Fairness cuts both ways, so to be explicit about what from this week’s debate does not transfer to the Splunk-bound logging question:

Duggan’s maintainer-concentration data is about the language SDKs, not the Collector. The Collector and its Splunk distribution are actively maintained, and prebuilt binaries mean most users never touch the Collector Builder.
Most performance complaints in the thread target SDK instrumentation overhead in application processes. An agent reading log files off disk - OTel’s filelog receiver or Collectord alike - has a different, far tamer profile.
“Every vendor’s OTel support is alpha/beta” is about APM vendors ingesting OTLP. Splunk HEC ingestion of collector output is mature.

The critique that does transfer is the one the thread kept circling: general-purpose abstractions charge a complexity tax, and the tax is worth paying only when you use the generality. A pipeline that can route any signal to any backend is the right tool when you have many signals and many backends. When the destination is Splunk and the payload is cluster telemetry, you are configuring receivers, processors, exporters, and pipelines to reimplement what a purpose-built agent does with an annotation.

The decision in one paragraph

Run the OTel Collector for traces - alongside either choice. Choose it for logs too if you fan out to backends only OTLP reaches, if an open-source-only policy binds you, or if your team already operates collector pipelines fluently and likes them. Choose Collectord when the destination is Splunk - or Elasticsearch, OpenSearch, or a syslog SIEM like QRadar, all of which it also speaks - and you want working dashboards on day one, per-workload self-service for routing, filtering, masking, sampling, and throttling, sidecar-free collection of the log files your apps write to volumes, multi-tenant isolation via CRDs, FIPS and Red Hat certified images, and a vendor who investigates your issue and ships the patch. The full comparison table covers both, row by row, next to the deprecated Splunk Connect for Kubernetes.

And if you are mid-debate about OTel complexity in your own team: the debate is healthy, but it resolves per-job, not per-ideology. Pick the general-purpose tool where you need the generality, the purpose-built tool where you don’t, and let them coexist - ours do.

Evaluating the switch? Start a 30-day trial - installation to rendered dashboards is about ten minutes, which is also the fastest way to see the difference this post describes.

Collectord vs the OpenTelemetry Collector for Kubernetes logs in Splunk