Monitoring Linux

Troubleshooting

Verify configuration

When data isn’t showing up in Splunk, the first thing to run is collectord verify. It exercises every input and output declared in your configuration — license check, HEC connectivity, file paths, journald, proc, and cgroup access — and prints OK or the exact error for each.

bash
1sudo /opt/collectorforlinux/bin/collectord verify --environment linux --conf /opt/collectorforlinux/etc

A healthy run looks like this:

text
 1...
 2Version = 5.12.270
 3Build date = 191031
 4Environment = linux
 5
 6 
 7  General:
 8  + conf: OK
 9  + db: OK
10  + db-meta: OK
11  + instanceID: OK
12    instanceID = 2N9ERP0D9SANAPL56IOQNBCJH0
13  + license load: OK
14  + license expiration: OK
15  + license connection: OK
16 
17  Splunk output:
18  + OPTIONS(url=https://127.0.0.1:8088/services/collector/event/1.0): OK
19  + POST(url=https://127.0.0.1:8088/services/collector/event/1.0, index=): OK
20 
21  File Inputs:
22  + input(syslog): OK
23    path /var/log/
24  + input(logs): OK
25    path /var/log/
26 
27  System Input:
28  + path cgroup: OK
29  + path proc: OK
30 
31  Network stats Input:
32  + path proc: OK
33 
34  Network socket table Input:
35  + path proc: OK
36 
37  Proc Input:
38  + path proc: OK
39 
40  Mount Input:
41  + stats: OK
42 
43  Journald input:
44  + input(journald): OK

Any line that’s not OK points at the problem — wrong HEC URL, missing token, unreadable path, expired license, blocked outbound traffic to the licensing endpoint. Fix it, rerun verify, and restart the service.