Troubleshooting - Monitoring Kubernetes

Find out why Collectord’s pods don’t start or Splunk shows no data, and collect what support needs to help. Start with collectord verify, then look for your symptom below.

Verify configuration

When something looks off, first run collectord verify from inside a Collectord pod. It checks the configuration end to end and reports each item as OK or FAILED: the license, the Splunk output, the container runtime, file inputs and Prometheus endpoints.

Start by listing the Collectord pods:

bash

1$ kubectl get pods -n collectorforkubernetes
2NAME                                           READY     STATUS    RESTARTS   AGE
3collectorforkubernetes-addon-857fccb8b9-t9qgq   1/1       Running   1          1h
4collectorforkubernetes-bwmwr                    1/1       Running   0          1h
5collectorforkubernetes-xbnaa                    1/1       Running   0          1h

Collectord runs as two workloads: a DaemonSet (collectorforkubernetes) and a single Deployment add-on (collectorforkubernetes-addon). To exercise every code path, run verify against three pods:

the add-on;
a DaemonSet pod on a control-plane node, which scrapes the control plane;
a DaemonSet pod on a worker.

kubectl get pods -n collectorforkubernetes -o wide shows which node each pod runs on.

bash

1$ kubectl exec -n collectorforkubernetes collectorforkubernetes-addon-857fccb8b9-t9qgq -- /collectord verify
2$ kubectl exec -n collectorforkubernetes collectorforkubernetes-bwmwr -- /collectord verify
3$ kubectl exec -n collectorforkubernetes collectorforkubernetes-xbnaa -- /collectord verify

Before 26.10, a second DaemonSet, collectorforkubernetes-master, ran on the control-plane nodes.

Each command logs a few INFO lines while it loads the configuration, then prints a report. On a DaemonSet pod of a control-plane node, the report looks like this:

text

 1Version = 26.10.1
 2Build date = 261015
 3Commit = a14264c61d5e6177298e2fbfd6a56ae826833634
 4Environment = kubernetes
 5
 6
 7  DNS:
 8    resolver = go (nothing in the environment, /etc/resolv.conf or /etc/nsswitch.conf asks for the C library)
 9  + lookup(host=hec.example.com): OK
10    for = [output.splunk] url
11    took = 1ms, addresses = 192.0.2.10
12
13  General:
14  + conf: OK
15  + db: OK
16  + db-meta: OK
17  + instanceID: OK
18    instanceID = 2LEKCFD4KT4MUBIAQSUG7GRSAG
19  + license load: OK
20  + license expiration: OK
21  + license connection: OK
22
23  Splunk output:
24  + OPTIONS(url=https://hec.example.com:8088/services/collector/event/1.0): OK
25  + POST(url=https://hec.example.com:8088/services/collector/event/1.0, index=) (TLSVersion=TLS 1.3): OK
26
27  Kubernetes configuration:
28  + api: OK
29  + pod cgroup: OK
30    pods = 9
31  + container cgroup: OK
32    containers = 18
33  + volumes root: OK
34  + runtime: OK
35    containerd
36
37  CRI-O configuration:
38  - ignored: OK
39    kubernetes uses other container runtime
40
41  Containerd configuration:
42  + api: OK
43  + files: OK
44
45  File Inputs:
46  x input(audit-logs): FAILED
47    cannot access /rootfs/var/log/kube-apiserver/ (err = stat /rootfs/var/log/kube-apiserver/: no such file or directory)
48  + input(logs): OK
49    path /rootfs/var/log/
50
51  System Input:
52  + path cgroup: OK
53  + path proc: OK
54
55  Network stats Input:
56  + path proc: OK
57
58  Network socket table Input:
59  + path proc: OK
60
61  Proc Input:
62  + path proc: OK
63
64  Mount Input:
65  + stats: OK
66    path /rootfs/var/lib/kubelet/
67
68  Prometheus input:
69  + input(controller-manager) pod kube-system/kube-controller-manager-control-plane-1: OK
70    4109 metrics from https://127.0.0.1:10257/metrics
71  + input(coredns) pod kube-system/coredns-559f6c778d-ftl87: OK
72    218 metrics from http://10.244.0.2:9153/metrics
73  + input(coredns) pod kube-system/coredns-559f6c778d-xsnj7: OK
74    218 metrics from http://10.244.0.3:9153/metrics
75  + input(etcd) pod kube-system/etcd-control-plane-1: OK
76    1810 metrics from http://127.0.0.1:2381/metrics
77  + input(kube-proxy): OK
78    854 metrics from http://127.0.0.1:10249/metrics
79  + input(kubelet): OK
80    2024 metrics from https://192.0.2.21:10250/metrics
81  + input(kubernetes-api) pod kube-system/kube-apiserver-control-plane-1: OK
82    41780 metrics from https://127.0.0.1:6443/metrics
83  + input(scheduler) pod kube-system/kube-scheduler-control-plane-1: OK
84    3874 metrics from https://127.0.0.1:10259/metrics
85
86  Journald input:
87  + input(journald): OK
88
89
90Errors: 1

When a check fails, the report ends with the number of failures. Not every failure is a real problem, and the one above is harmless.

input(audit-logs) reads the audit log files under /var/log/kube-apiserver/, and this cluster’s API server keeps none there. It writes its audit log to standard output, as Audit logs sets it up, and Collectord forwards that with the other container logs. A cluster that doesn’t audit at all fails this check the same way.

An input that doesn’t apply to a node counts no error. On a worker node, the API server audit-log input is listed as not read on this node. A control-plane Prometheus input that finds no pod of its component on the node is listed as N/A:

text

1  File Inputs:
2  - input(audit-logs): not read on this node
3    its labels do not match nodeSelector "node-role.kubernetes.io/control-plane"
4  + input(logs): OK
5    path /rootfs/var/log/
6...
7  Prometheus input:
8  - input(controller-manager): N/A
9    no pod in kube-system matching component=kube-controller-manager or k8s-app=kube-controller-manager runs on this node

Inputs that ship disabled, such as [input.files::syslog], aren’t checked. The add-on runs none of the node inputs, so its report ends after the Kubernetes configuration.

After you fix a real configuration error, such as a wrong Splunk URL, kubectl apply -f ./collectorforkubernetes.yaml won’t restart the running pods. Delete them, and the workloads recreate them with the new configuration: kubectl delete pods --all -n collectorforkubernetes.

Describe command

Annotations can come from namespaces, workloads, configurations and pods, so it can be hard to track which ones apply to a pod or container. The collectord describe command shows which annotations a specific pod uses. Run it in any Collectord pod on the cluster:

bash

1kubectl exec -n collectorforkubernetes collectorforkubernetes-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres

Since version 26.04, the describe command also tags each resolved field with its origin, in square brackets:

[pod]: the value comes from a pod annotation.
[namespace]: the value comes from a namespace annotation.
[configuration:<name>]: the value comes from a Collectord CRD Configuration resource, and <name> matches the resource name.

When the same annotation is defined at several levels, the tags show which level of the configuration hierarchy wins. For example, a pod-level annotation may override a CRD-level default, or a namespace annotation may route logs to an unexpected output:

bash

1$ kubectl exec -n collectorforkubernetes collectorforkubernetes-fqhmv -- /collectord describe --namespace webportal --pod audit-logger-774675c89c-rpfwx | grep '\['
2logs-type [pod] = audit_logs
3volume.1-logs-name [pod] = data
4volume.1-logs-glob [pod] = *.log

Use it to debug why a pod routes to an unexpected output, uses the wrong sourcetype, or picks up a field extraction you didn’t expect.

Collect diagnostic information

When you open a support case, attach a diagnostic bundle, so we can reproduce the issue without a back-and-forth. The bundle includes memory and telemetry metrics, host Linux information, and the Collectord configuration, with the Splunk URL and HEC token stripped out. It includes performance profiles when you ask for them.

Run all four steps below.

1. Collect the internal diag information

Pick any Collectord pod and run collectord diag. The command takes a few minutes:

bash

1kubectl exec -n collectorforkubernetes collectorforkubernetes-bwmwr -- /collectord diag --stream 1>diag.tar.gz

You can extract the archive yourself to see exactly what’s in it: basic telemetry metrics, host Linux info and license metadata, and the performance and memory profiles when you ask for them.

If you run kubectl from PowerShell older than 7.4, including Windows PowerShell 5.1, the --stream command above produces a corrupted archive. PowerShell re-encodes the redirected output with its default text encoding, which breaks the gzip stream. The shell is what matters, not the operating system: bash, zsh and cmd.exe pass the bytes through unchanged.

On Windows, run the same command through cmd.exe:

powershell

1cmd /c "kubectl exec -n collectorforkubernetes collectorforkubernetes-bwmwr -- /collectord diag --stream > diag.tar.gz"

Or upgrade to PowerShell 7.4 or later, which preserves the byte stream, and run the original command.

Since 5.20.400, performance profiles aren’t collected by default: add --include-performance-profiles if you need them. Since 5.24, two more flags are available: --quiet suppresses the progress output on stderr, and --keep keeps the diag file in Collectord’s data directory after streaming it.

2. Collect logs

bash

1kubectl logs -n collectorforkubernetes --timestamps collectorforkubernetes-bwmwr  1>collectorforkubernetes.log 2>&1

3. Run verify

bash

1kubectl exec -n collectorforkubernetes collectorforkubernetes-bwmwr -- /collectord verify > verify.log

4. Prepare tar archive

bash

1tar -czvf collectorforkubernetes-$(date +%s).tar.gz verify.log collectorforkubernetes.log diag*.tar.gz

Pod is not getting scheduled

If Collectord pods never appear, the DaemonSet isn’t placing them on any node. Check the desired and current counts:

bash

1kubectl get daemonset --namespace collectorforkubernetes

Zeros under CURRENT, READY, or UP-TO-DATE mean the controller couldn’t create a single pod:

text

1NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
2collectorforkubernetes   3         0         0       0            0           kubernetes.io/os=linux   1m

Describe the DaemonSet to see the underlying reason:

bash

1$ kubectl describe daemonsets --namespace collectorforkubernetes

The Events section at the bottom tells you what failed:

text

1...
2Events:
3  Type     Reason            Age                From                  Message
4  ----     ------            ----               ----                  -------
5  Warning  FailedCreate      31m                daemonset-controller  Error creating: pods "collectorforkubernetes-6xrkd" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true), ...

That error means Pod Security Admission enforces the baseline or restricted level in the namespace. The namespace has to admit privileged pods, because of what Collectord’s workloads do:

The DaemonSet reads the node’s filesystem, journal and cgroup tree through a host mount, runs on the host network, and is privileged.
The addon Deployment mounts, from the host, the state directory it shares with the DaemonSet.

Label the namespace to enforce the privileged level:

bash

1kubectl label namespace collectorforkubernetes pod-security.kubernetes.io/enforce=privileged --overwrite

Failed to pull the image

If the DaemonSet shows pods but READY is below DESIRED, the kubelet is probably failing to pull the image:

bash

1$ kubectl get daemonsets --namespace collectorforkubernetes

text

1NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
2collectorforkubernetes   1         1         0       1            0           kubernetes.io/os=linux   6m

List the pods to confirm:

bash

1$ kubectl get pods --namespace collectorforkubernetes

A status of ImagePullBackOff confirms the kubelet can’t reach the registry:

text

1NAME                           READY   STATUS             RESTARTS   AGE
2collectorforkubernetes-55t61   0/1     ImagePullBackOff   0          2m

The nodes can’t pull the image, registry.outcold.solutions/outcoldsolutions/collectorforkubernetes: they can’t reach the registry, or the ghcr.io and pkg-containers.githubusercontent.com hosts its pulls go through. If your nodes can reach Docker Hub instead, the same image is there as docker.io/outcoldsolutions/collectorforkubernetes.

Describe the pods to see the kubelet’s actual error:

bash

1$ kubectl describe pods --namespace collectorforkubernetes

The Events section shows the pull attempts:

text

1Events:
2  Type     Reason   Age                From     Message
3  ----     ------   ----               ----     -------
4  Normal   Pulling  2m (x4 over 3m)    kubelet  Pulling image "registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1"
5  Normal   BackOff  1m (x6 over 3m)    kubelet  Back-off pulling image "registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1"

Blocked access to external registries

If your cluster can’t reach registry.outcold.solutions or Docker Hub for security reasons, mirror the image to an internal registry from a host that does have outbound access.

Copying the image to your own registry

Pull the image:

bash

1$ docker pull registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1

Re-tag it under your registry:

bash

1docker tag  registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1 [YOUR_REGISTRY]/outcoldsolutions/collectorforkubernetes:26.10.1

Push it:

bash

1docker push [YOUR_REGISTRY]/outcoldsolutions/collectorforkubernetes:26.10.1

Then point the manifest at your registry:

yaml

1image: [YOUR_REGISTRY]/outcoldsolutions/collectorforkubernetes:26.10.1

Keep the image’s name, collectorforkubernetes, under any registry path, tag or digest. Monitoring Kubernetes finds Collectord’s own log by that name, or by the container’s name, collectorforkubernetes. If you rename the image and its container has another name too, add its source to macro_kubernetes_logs_collectord and to the app’s props.conf.

If you need to move the image between hosts that can’t talk to each other, save it to a tar:

bash

1$ docker image save registry.outcold.solutions/outcoldsolutions/collectorforkubernetes:26.10.1 > collectorforkubernetes.tar

Then load it on the other host:

bash

1$ cat collectorforkubernetes.tar | docker image load

Pod is crashing or running, but you don’t see any data

Get the Pod information

When a pod is crash-looping, the most useful thing in its YAML is lastState: it tells you why the previous container exited. Replace the pod name with the one that’s crashing:

bash

1kubectl get pod -n collectorforkubernetes -o yaml collectorforkubernetes-mshxd

If lastState looks like this:

yaml

1lastState:
2  terminated:
3    containerID: containerd://8e9086aaf65b86d6d070f98ef4c5c59d9c838401a1f40765dd997723144d65db
4    exitCode: 128
5    finishedAt: "2022-10-16T05:58:13Z"
6    message: path / is mounted on / but it is not a shared or slave mount
7    reason: ContainerCannotRun
8    startedAt: "2022-10-16T05:58:13Z"

Then the host’s root filesystem isn’t a shared or slave mount, so Collectord can’t propagate mounts back. In collectorforkubernetes.yaml, find every mountPropagation: HostToContainer and comment it out. The only feature you lose is the auto-discovery of volumes containing application logs.

Email us at support@outcoldsolutions.com and we’ll help you configure it properly.

Check Collectord logs

The Collectord logs themselves usually tell you what’s going wrong. A healthy start of a DaemonSet pod on a worker node looks like this:

text

1$ kubectl logs collectorforkubernetes-gvhgw --namespace collectorforkubernetes
2INFO 2026/10/15 16:54:04.738097 outcoldsolutions.com/collectord/internal/daemon/daemon.go:91: Build date = 261015, version = 26.10.1, commit = a14264c61d5e6177298e2fbfd6a56ae826833634
3INFO 2026/10/15 16:54:04.789368 outcoldsolutions.com/collectord/environment/logs.go:40: [input.files::audit-logs] not read on this node: its labels do not match nodeSelector "node-role.kubernetes.io/control-plane"
4INFO 2026/10/15 16:54:04.789399 outcoldsolutions.com/collectord/pipeline/input/file/dir/watcher.go:237: logs - watching /rootfs/var/log/ (glob = "", match = "^(([\\w\\-.]+\\.log(.[\\d\\-]+)?)|(docker))$", exclude = "^(auth|cron|daemon|kern|lpr|mail|user|ufw)\\.log(.[\\d\\-]+)?$")
5INFO 2026/10/15 16:54:04.790850 outcoldsolutions.com/collectord/pipeline/input/file/dir/watcher.go:859: logs - added file /rootfs/var/log/alternatives.log
6INFO 2026/10/15 16:54:04.792501 outcoldsolutions.com/collectord/pipeline/input/file/dir/watcher.go:859: logs - added file /rootfs/var/log/dpkg.log
7INFO 2026/10/15 16:54:04.987276 outcoldsolutions.com/collectord/pipeline/input/kube/watcher/source.go:171: pods client created
8INFO 2026/10/15 16:54:05.014347 outcoldsolutions.com/collectord/pipeline/watcher/watcher.go:399: kubernetes_watcher - watching 08c48d0f-4465-4e81-80a1-f89cf7b30eab
9INFO 2026/10/15 16:54:05.019127 outcoldsolutions.com/collectord/pipeline/watcher/watcher.go:399: kubernetes_watcher - watching 643705ea-7a7f-46bf-b18d-b0f1dda2bc59

Each kubernetes_watcher - watching line is a pod on the node, by its UID, whose logs and metrics Collectord collects from then on.

If you forget to set url or token for the Splunk output, Collectord refuses to start, and the last line of its log names what’s missing:

text

1ERROR 2026/10/15 16:54:04.740625 outcoldsolutions.com/collectord/main.go:98: collectord monitor: url (or urls) are required

If the license server is unreachable, the logs say so. For air-gapped clusters, contact us for a license that doesn’t require internet access.

If Splunk itself is unreachable, the logs say that too, and Collectord keeps retrying:

text

1WARN 2026/10/15 17:04:55.127045 outcoldsolutions.com/collectord/pipeline/output/httpout/delivery.go:579: output="splunk" thread=1 failed to post a batch of 1 records for partition "": Post "https://hec.example.com:8088/services/collector/event/1.0": dial tcp 192.0.2.10:8088: connect: connection refused — verify the connection; retrying with backoff (attempt 3)

If the logs are clean but the Monitoring Kubernetes app is empty, the most common cause is that your HEC token writes to a non-default index that the Splunk role can’t search. There are two fixes:

Search the index by default. Add the index as a default index for the role.
Scope the macros to your index. The macros live in Splunk Web under Settings > Advanced Search > Search Macros. Every events-index macro, macro_kubernetes_logs included, builds on macro_kubernetes_base, so set your index there once, for example (index=your_index). Every dashboard is built on these macros, so the change takes effect immediately.

See Splunk indexes.