Troubleshooting

Verify configuration

The first thing to do when something looks off is to run collectord verify inside the Collectord container. It checks the configuration end-to-end - license, Splunk output, Docker connection, file inputs - and reports each item as OK or FAILED.

1docker exec collectorfordocker /collectord verify

The output looks like this:

 1...
 2Version = 5.2.176
 3Build date = 181012
 4Environment = docker
 5
 6
 7  General:
 8  + conf: OK
 9  + db: OK
10  + db-meta: OK
11  + instanceID: OK
12    instanceID = 2LCKRSKBSSO30B8USER5DVQS30
13  + license load: OK
14    trial
15  + license expiration: OK
16    license expires 2018-10-30 22:27:31.8491662 +0000 UTC
17  + license connection: OK
18
19  Splunk output:
20  + OPTIONS(url=https://host.docker.internal:8088/services/collector/event/1.0): OK
21  + POST(url=https://host.docker.internal:8088/services/collector/event/1.0, index=): OK
22
23  Docker configuration:
24  + connect: OK
25    containers = 4
26  + path: OK
27  + cgroup: OK
28    containers = 4
29  + files: OK
30
31  File Inputs:
32  x input(syslog): FAILED
33    no matches
34  + input(logs): OK
35    path /rootfs/var/log/
36
37  System Input:
38  + path cgroup: OK
39  + path proc: OK
40
41  Network stats Input:
42  + path proc: OK
43
44  Network socket table Input:
45  + path proc: OK
46
47  Proc Input:
48  + path proc: OK
49
50  Mount Input:
51  + stats: OK
52
53
54Errors: 1

The total number of errors appears at the bottom. Not every failure is a real problem - in this example, input(syslog) failed because the host doesn’t have any syslog files to forward, which is fine.

Describe command

When you’ve configured forwarding through container labels and want to confirm what Collectord actually resolved for a given container, use collectord describe. It prints every field in effect for that container - the source type, output, fields, log paths, and so on:

1docker exec collectorfordocker /collectord describe --container intelligent_bose

This is especially useful when debugging why a container is routing to an unexpected output, using the wrong sourcetype, or picking up a field extraction you didn’t expect.

Collect diagnostic information

When you open a support case, attach a diagnostic bundle so we can reproduce the issue without a back-and-forth. The bundle includes performance profiles, memory and telemetry metrics, host Linux information, and the Collectord configuration - Splunk URL and HEC token are stripped out.

1. Collect internal diag information from Collectord instance run following command

Run collectord diag inside the container - the command takes a few minutes:

1docker exec collectorfordocker /collectord diag --stream 1>diag.tar.gz

You can extract the archive yourself to see exactly what’s in it - performance and memory profiles, basic telemetry metrics, host Linux info, and license metadata.

Since 5.20.400, performance profiles aren’t collected by default. Add --include-performance-profiles if you need them.

Since 5.24, two more flags are available: --quiet suppresses stdout output, and --keep writes the diag file to Collectord’s data directory instead of streaming it.

If you run docker from PowerShell older than 7.4, including Windows PowerShell 5.1, the --stream command above produces a corrupted archive: PowerShell re-encodes the redirected output with its default text encoding and breaks the gzip stream. The shell is what matters, not the operating system - bash, zsh, and cmd.exe pass the bytes through unchanged.

On Windows, run the same command through cmd.exe:

1cmd /c "docker exec collectorfordocker /collectord diag --stream > diag.tar.gz"

Or upgrade to PowerShell 7.4 or later, which preserves the byte stream, and run the original command.

2. Collect logs

1docker logs --details --timestamps collectorfordocker  1>collectorfordocker.log 2>&1

3. Run verify

1docker exec collectorfordocker /collectord verify > verify.log

4. Prepare tar archive

1tar -czvf collectorfordocker-$(date +%s).tar.gz verify.log collectorfordocker.log diag.tar.gz