Concepts

A short orientation to how the ElasticSearch / OpenSearch integration actually works. Read this once and the rest of the docs will make sense without piecing things together from installation, configuration, and annotations all at once.

How it’s structured

This is a forwarder-only product - there’s no Splunk-style app on the receiving end:

• Collectord - the agent. Runs as a DaemonSet, control-plane nodes included, plus a single Deployment add-on that watches the Kubernetes API server for events and the objects that Collectord ships out of the box (Pods, Deployments, Nodes and ResourceQuotas). Same Collectord binary as the Splunk product; only the output is different.
• ElasticSearch / OpenSearch - the destination. Collectord ships an index template for logs-collectord-${COLLECTORD_VERSION} and a paired -failed index for events that fail ingestion (typically a mapping conflict). On ElasticSearch it also ships an index lifecycle policy, logs-collectord, which rolls the datastream over every 30 days or at 50 GB per primary shard and deletes nothing. Adjust the JSON files in the configuration to match your retention and shard layout.

If you’re already running Collectord for Splunk on the same cluster from its manifest, the ElasticSearch manifest deploys into the same collectorforkubernetes namespace (a Helm release goes into the namespace you install it into) - see the Installation note about co-existence, including which manifest to apply last.

What Collectord forwards

Collectord forwards logs and events; it does not forward metrics or Prometheus data through this integration. The ElasticSearch / OpenSearch product is logs-focused.

The categories you’ll see in your indices:

• Logs - container stdout / stderr, application logs from files inside the container or on a mounted volume, host logs (the systemd journal and the *.log files under /var/log), and the API server’s audit logs. [input.files::audit-logs] reads audit*.log under /var/log/kube-apiserver/ on the control-plane nodes; the API server writes nothing there until you enable audit logging on it with --audit-log-path in that directory.
• Events - Kubernetes API events.
• Objects - Pod, Deployment, Node and ResourceQuota specs, streamed via watch. Other kinds (ConfigMaps, Secrets, StatefulSets, …) are opt-in via additional [input.kubernetes_watch::*] stanzas.

Before 26.10, the ElasticSearch and OpenSearch manifests read no audit logs, and their addon watched only Pods and Deployments.

Each input writes to a datastream named logs-collectord-${COLLECTORD_VERSION} by default; override per workload with annotations.

Inputs, outputs, and pipes

Internally, Collectord is a pipeline of three concepts:

• An input discovers data - [input.files::logs] reads /var/log, [input.kubernetes_events] watches the API server, [input.app_logs] picks up application log files exposed via annotations.
• A pipe transforms events on the way through - replace, hash, extract, override, sample, throttle.
• An output ships the result somewhere - [output.elasticsearch] is the default; devnull exists for dropping data.

Most of annotations is just “configure pipes and the destination datastream, scoped to one pod or namespace.”

Annotation prefix

The ElasticSearch configuration sets annotationsSubdomain = elasticsearch, so its annotations use the elasticsearch.collectord.io/ prefix instead of plain collectord.io/. The OpenSearch configuration sets annotationsSubdomain = opensearch, so its annotations use opensearch.collectord.io/. This is so you can run a Splunk-output Collectord and an ElasticSearch- or OpenSearch-output Collectord on the same cluster without their annotations colliding - each only reads its own namespace.

Before 26.10, the OpenSearch manifest read elasticsearch.collectord.io/ annotations (the OpenSearch Helm chart already read opensearch.collectord.io/). See Upgrade from 26.04 to 26.10.

If you want an annotation to apply to every Collectord instance regardless of subdomain, use the collectord.collectord.io/ prefix.

Where configuration comes from

The same setting can be specified in several places. The general layering, highest priority first:

1. Pod annotations - most specific.
2. Workload annotations - annotations on the Deployment / StatefulSet / DaemonSet propagate to its pods.
3. Namespace annotations - apply to every pod in the namespace.
4. CRD Configuration - cluster-level defaults applied by selector.
5. ConfigMap files - 001-general.conf and the role-specific files mounted into the relevant Collectord pods. The lowest-priority layer.

A CRD Configuration can be marked with force: true to override pod, workload, and namespace annotations - see Annotations → Forcing Cluster Level Annotations.

When the same setting is defined at multiple levels and the wrong value is winning, run collectord describe from inside a Collectord pod - see Troubleshooting → Describe.

Auto-discovery vs explicit opt-in

Some things Collectord finds on its own; others require you to point at them:

If something you expect isn’t in your index, the first question is usually: was this auto-discovered, or do I need to point Collectord at it?

How data flows

1Your container ─┐
2                ├─► [input] ──► [pipe] ──► [output.elasticsearch] ──► ES / OpenSearch
3Your annotation ┘                                                           │
4                                                                            ▼
5                                                                  Kibana / OpenSearch Dashboards

Collectord runs on the nodes, reads from local sources (filesystem, the API server) and pushes events to your ElasticSearch or OpenSearch HTTP API. Nothing on the receiving side is product-specific - once the events are indexed you query them with whatever tooling you use today.

Where to go next

• New cluster: Installation.
• Tuning what gets forwarded for a specific workload: Annotations.
• Something looks wrong: Troubleshooting.