Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Pricing Contact Us Blog Evaluation licenses Partners License Agreement Privacy Policy FAQ

Monitoring Docker

Troubleshooting

Verify configuration

Available from collectorfordocker v5.2

Execute verify command to verify configuration.

1docker exec collectorfordocker /collectord verify

The output will look similar to

 1...
 2Version = 5.2.176
 3Build date = 181012
 4Environment = docker
 5
 6
 7  General:
 8  + conf: OK
 9  + db: OK
10  + db-meta: OK
11  + instanceID: OK
12    instanceID = 2LCKRSKBSSO30B8USER5DVQS30
13  + license load: OK
14    trial
15  + license expiration: OK
16    license expires 2018-10-30 22:27:31.8491662 +0000 UTC
17  + license connection: OK
18
19  Splunk output:
20  + OPTIONS(url=https://host.docker.internal:8088/services/collector/event/1.0): OK
21  + POST(url=https://host.docker.internal:8088/services/collector/event/1.0, index=): OK
22
23  Docker configuration:
24  + connect: OK
25    containers = 4
26  + path: OK
27  + cgroup: OK
28    containers = 4
29  + files: OK
30
31  File Inputs:
32  x input(syslog): FAILED
33    no matches
34  + input(logs): OK
35    path /rootfs/var/log/
36
37  System Input:
38  + path cgroup: OK
39  + path proc: OK
40
41  Network stats Input:
42  + path proc: OK
43
44  Network socket table Input:
45  + path proc: OK
46
47  Proc Input:
48  + path proc: OK
49
50  Mount Input:
51  + stats: OK
52
53
54Errors: 1

The output shows you a number of errors in the configuration. In this example there are no syslog files, so collector will not be able to forward these files.

Describe command

Available from collectorfordocker v5.12

You can use a describe command of collectord to get information which annotations are used for the specific container.

1docker exec collectorfordocker /collectord describe --container intelligent_bose

Collect diagnostic information

If you need to open a support case you can collect diagnostic information, including performance, metrics and configuration (excluding splunk URL and Token).

1. Collect internal diag information from Collectord instance run following command

Available from collectorfordocker v5.2

The following command takes several minutes.

1docker exec collectorfordocker /collectord diag --stream 1>diag.tar.gz

You can extract a tar archive to verify the information that we collect. We include information about performance, memory usage, basic telemetry metrics, information file with the information of the host Linux version and basic information about the license.

Since 5.20.400 performance information is not collected by default unless you include a flag --include-performance-profiles in the command.

2. Collect logs

1docker logs --details --timestamps collectorfordocker  1>collectorfordocker.log 2>&1

3. Run verify

Available from collectorfordocker v5.2

1docker exec collectorfordocker /collectord verify > verify.log

4. Prepare tar archive

1tar -czvf collectorfordocker-$(date +%s).tar.gz verify.log collectorfordocker.log diag.tar.gz

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.

Red Hat
Splunk
AWS