Troubleshooting

Verify configuration

The first thing to do when something looks off is to run collectord verify from inside a Collectord pod. It checks the configuration end-to-end - license, ElasticSearch output, container runtime, file inputs, journald - and reports each item as OK or FAILED.

Start by listing the Collectord pods:

1$ kubectl get pods -n collectorforkubernetes
2NAME                                           READY     STATUS    RESTARTS   AGE
3collectorforkubernetes-elasticsearch-addon-857fccb8b9-t9qgq   1/1       Running   1          1h
4collectorforkubernetes-elasticsearch-bwmwr                    1/1       Running   0          1h
5collectorforkubernetes-elasticsearch-xbnaa                    1/1       Running   0          1h

Collectord runs as two workloads - a DaemonSet and a single Deployment add-on. Run verify against the add-on, a DaemonSet pod on a control-plane node (it reads the API server’s audit logs) and one on a worker, so every code path is exercised (kubectl get pods -n collectorforkubernetes -o wide shows which node each pod runs on):

1$ kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-addon-857fccb8b9-t9qgq -- /collectord verify
2$ kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-bwmwr -- /collectord verify
3$ kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-xbnaa -- /collectord verify

Each command prints a report that starts with Version = 26.10.1 and the environment, then one section per area with one line per check: + for OK, x for FAILED, and - for a check that doesn’t apply. The output and file-input sections of a DaemonSet pod on a worker node read:

1  ElasticSearch output:
2  + OPTIONS(url=https://elasticsearch.example.com:9200): OK
3
4  File Inputs:
5  - input(audit-logs): not read on this node
6    its labels do not match nodeSelector "node-role.kubernetes.io/control-plane"
7  + input(logs): OK
8    path /rootfs/var/log/

The API server’s audit logs are on the control-plane nodes, so a worker doesn’t read them, and that line counts no error. On a control-plane node the line reads + input(audit-logs): OK, or x input(audit-logs): FAILED until the API server writes its audit logs to /var/log/kube-apiserver/.

When a check fails, the report ends with the number of failures, such as Errors: 1.

If you fix a real configuration error, kubectl apply -f ./collectorforkubernetes-elasticsearch.yaml won’t restart the running pods. Delete them so the workloads recreate them with the new config: kubectl delete pods --all -n collectorforkubernetes.

Describe command

When the same annotation is defined at the namespace, workload, CRD Configuration, and pod level, it can be hard to tell which value Collectord actually uses for a given container. The collectord describe command resolves the full set of annotations for a specific pod and container so you can see exactly what’s in effect. You can run it from any Collectord pod:

1kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres

Starting with version 26.04, the describe command also tags each resolved field with its origin in square brackets:

• [pod] - the value comes from a pod annotation
• [namespace] - the value comes from a namespace annotation
• [configuration:<name>] - the value comes from a Collectord CRD Configuration resource (the <name> matches the resource name)

This makes it easy to trace which level of the configuration hierarchy is winning when the same annotation (with the elasticsearch.collectord.io/ prefix) is defined at multiple levels - for example, when a CRD-level default is being overridden by a pod-level annotation, or when a namespace annotation is unexpectedly routing logs to a different datastream:

1$ kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-fqhmv -- /collectord describe --namespace webportal --pod audit-logger-774675c89c-rpfwx | grep '\['
2logs-type [pod] = audit_logs
3volume.1-logs-name [pod] = data
4volume.1-logs-glob [pod] = *.log

This is especially useful when debugging why a pod is routing to an unexpected datastream, using the wrong index, or picking up a field extraction you didn’t expect.

Collect diagnostic information

When you open a support case, attach a diagnostic bundle so we can reproduce the issue without a back-and-forth. The bundle includes performance profiles, memory and telemetry metrics, host Linux information, and the Collectord configuration - the ElasticSearch URL and credentials are stripped out.

Run all four steps below.

1. Collect diagnostics information run following command

Pick any Collectord pod and run collectord diag. The command takes a few minutes:

1kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-bwmwr -- /collectord diag --stream 1>diag.tar.gz

You can extract the archive yourself to see exactly what’s in it - performance and memory profiles, basic telemetry metrics, host Linux info, and license metadata.

Performance profiles aren’t collected by default. Add --include-performance-profiles if you need them.

2. Collect logs

1kubectl logs -n collectorforkubernetes --timestamps collectorforkubernetes-elasticsearch-bwmwr  1>collectorforkubernetes.log 2>&1

3. Run verify

1kubectl exec -n collectorforkubernetes collectorforkubernetes-elasticsearch-bwmwr -- /collectord verify > verify.log

4. Prepare tar archive

1tar -czvf collectorforkubernetes-$(date +%s).tar.gz verify.log collectorforkubernetes.log diag.tar.gz