Troubleshooting

Run verify command and diagnose common issues

Find out why a Linux host’s data isn’t reaching Splunk, and collect what support needs when you open a case.

Verify configuration

When data isn’t showing up in Splunk, the first thing to run is collectord verify. It exercises every input and output declared in your configuration: license check, HEC connectivity, file paths, journald, proc, and cgroup access. For each, it prints OK or the exact error.

bin/collectord runs the build for your host’s architecture and reads the configuration in /opt/collectorforlinux/etc.

bash

1sudo /opt/collectorforlinux/bin/collectord verify

A healthy run looks like this:

text

 1...
 2Version = 26.10.1
 3Build date = 261015
 4Commit = a14264c61d5e6177298e2fbfd6a56ae826833634
 5Environment = linux
 6
 7 
 8  DNS:
 9    resolver = go (nothing in the environment, /etc/resolv.conf or /etc/nsswitch.conf asks for the C library)
10  + lookup(host=splunk.example.com): OK
11    for = [output.splunk] url
12    took = 2ms, addresses = 192.0.2.10
13  + lookup(host=license.outcold.solutions): OK
14    for = [general] telemetryEndpoint
15    took = 28ms, addresses = 198.51.100.20
16 
17  General:
18  + conf: OK
19  + db: OK
20  + db-meta: OK
21  + instanceID: OK
22    instanceID = 33DRTTEPSO5F76P6LFCFB81L20
23  + license load: OK
24  + license expiration: OK
25  + license connection: OK
26 
27  Splunk output:
28  + OPTIONS(url=https://splunk.example.com:8088/services/collector/event/1.0): OK
29  + POST(url=https://splunk.example.com:8088/services/collector/event/1.0, index=) (TLSVersion=TLS 1.3): OK 
30 
31  File Inputs:
32  + input(logs): OK
33    path /var/log/
34 
35  System Input:
36  + path cgroup: OK
37  + path proc: OK
38 
39  Network stats Input:
40  + path proc: OK
41 
42  Network socket table Input:
43  + path proc: OK
44 
45  Proc Input:
46  + path proc: OK
47 
48  Mount Input:
49  + stats: OK
50    mount / (ext4, /dev/vdb)
51 
52  Journald input:
53  + input(journald): OK
54 
55  Scripted inputs:
56  + script(package_updates): OK
57    command = /opt/collectorforlinux/scripts/package-updates.sh
58    user = nobody (uid 65534, gid 65534, home /nonexistent)

Any line that’s not OK points at the problem: a wrong HEC URL, a missing token, an unreadable path, an expired license, or blocked outbound traffic to the licensing endpoint. Fix it, rerun verify, and restart the service.

Collect diagnostic information

When you open a support case, attach a diagnostic bundle and Collectord’s log, so we can look at the issue without a back-and-forth. The bundle holds memory and telemetry metrics, host information and the Collectord configuration, with the Splunk URL and HEC token stripped out.

Collect the bundle, which takes a few minutes. Add --include-performance-profiles if support asks for them:
bash
```
1sudo /opt/collectorforlinux/bin/collectord diag --stream > diag.tar.gz
```

Collect Collectord’s log, which the service writes to the journal:

bash

1sudo journalctl -u collectorforlinux --since "-24h" > collectorforlinux.log

Collect the output of sudo /opt/collectorforlinux/bin/collectord verify (see Verify configuration).

Before 26.10, collectord diag on a Linux host signalled systemd instead of Collectord and ended with “timed out waiting for diag”.