Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Blog Contact Us Pricing Evaluation licenses Partners License Agreement Privacy Policy FAQ
Monitoring Docker

Troubleshooting

Verify configuration

Available since Collectord version 5.2

The first thing to do when something looks off is to run collectord verify inside the Collectord container. It checks the configuration end-to-end — license, Splunk output, Docker connection, file inputs — and reports each item as OK or FAILED.

bash
1docker exec collectorfordocker /collectord verify

The output looks like this:

text
 1...
 2Version = 5.2.176
 3Build date = 181012
 4Environment = docker
 5
 6
 7  General:
 8  + conf: OK
 9  + db: OK
10  + db-meta: OK
11  + instanceID: OK
12    instanceID = 2LCKRSKBSSO30B8USER5DVQS30
13  + license load: OK
14    trial
15  + license expiration: OK
16    license expires 2018-10-30 22:27:31.8491662 +0000 UTC
17  + license connection: OK
18
19  Splunk output:
20  + OPTIONS(url=https://host.docker.internal:8088/services/collector/event/1.0): OK
21  + POST(url=https://host.docker.internal:8088/services/collector/event/1.0, index=): OK
22
23  Docker configuration:
24  + connect: OK
25    containers = 4
26  + path: OK
27  + cgroup: OK
28    containers = 4
29  + files: OK
30
31  File Inputs:
32  x input(syslog): FAILED
33    no matches
34  + input(logs): OK
35    path /rootfs/var/log/
36
37  System Input:
38  + path cgroup: OK
39  + path proc: OK
40
41  Network stats Input:
42  + path proc: OK
43
44  Network socket table Input:
45  + path proc: OK
46
47  Proc Input:
48  + path proc: OK
49
50  Mount Input:
51  + stats: OK
52
53
54Errors: 1

The total number of errors appears at the bottom. Not every failure is a real problem — in this example, input(syslog) failed because the host doesn’t have any syslog files to forward, which is fine.

Describe command

Available since Collectord version 5.12

When you’ve configured forwarding through container labels and want to confirm what Collectord actually resolved for a given container, use collectord describe. It prints every field in effect for that container — the source type, output, fields, log paths, and so on:

bash
1docker exec collectorfordocker /collectord describe --container intelligent_bose

This is especially useful when debugging why a container is routing to an unexpected output, using the wrong sourcetype, or picking up a field extraction you didn’t expect.

Collect diagnostic information

When you open a support case, attach a diagnostic bundle so we can reproduce the issue without a back-and-forth. The bundle includes performance profiles, memory and telemetry metrics, host Linux information, and the Collectord configuration — Splunk URL and HEC token are stripped out.

1. Collect internal diag information from Collectord instance run following command

Available since Collectord version 5.2

Run collectord diag inside the container — the command takes a few minutes:

bash
1docker exec collectorfordocker /collectord diag --stream 1>diag.tar.gz

You can extract the archive yourself to see exactly what’s in it — performance and memory profiles, basic telemetry metrics, host Linux info, and license metadata.

Since 5.20.400, performance profiles aren’t collected by default. Add --include-performance-profiles if you need them.

2. Collect logs

bash
1docker logs --details --timestamps collectorfordocker  1>collectorfordocker.log 2>&1

3. Run verify

Available since Collectord version 5.2
bash
1docker exec collectorfordocker /collectord verify > verify.log

4. Prepare tar archive

bash
1tar -czvf collectorfordocker-$(date +%s).tar.gz verify.log collectorfordocker.log diag.tar.gz