Outcold Solutions LLC

Monitoring Docker - Version 5

Troubleshooting

Verify configuration

Available from collectorfordocker v5.2

Execute verify command to verify configuration.

docker exec collectorfordocker /collectord verify

The output will look similar to

...
Version = 5.2.176
Build date = 181012
Environment = docker


  General:
  + conf: OK
  + db: OK
  + db-meta: OK
  + instanceID: OK
    instanceID = 2LCKRSKBSSO30B8USER5DVQS30
  + license load: OK
    trial
  + license expiration: OK
    license expires 2018-10-30 22:27:31.8491662 +0000 UTC
  + license connection: OK

  Splunk output:
  + OPTIONS(url=https://host.docker.internal:8088/services/collector/event/1.0): OK
  + POST(url=https://host.docker.internal:8088/services/collector/event/1.0, index=): OK

  Docker configuration:
  + connect: OK
    containers = 4
  + path: OK
  + cgroup: OK
    containers = 4
  + files: OK

  File Inputs:
  x input(syslog): FAILED
    no matches
  + input(logs): OK
    path /rootfs/var/log/

  System Input:
  + path cgroup: OK
  + path proc: OK

  Network stats Input:
  + path proc: OK

  Network socket table Input:
  + path proc: OK

  Proc Input:
  + path proc: OK

  Mount Input:
  + stats: OK


Errors: 1

The output shows you a number of errors in the configuration. In this example there are no syslog files, so collector will not be able to forward these files.

Describe command

Available from collectorfordocker v5.12

You can use a describe command of collectord to get information which annotations are used for the specific container. 

docker exec collectorfordocker /collectord describe --container intelligent_bose

Collect diagnostic information

If you need to open a support case you can collect diagnostic information, including performance, metrics and configuration (excluding splunk URL and Token).

1. Collect internal diag information from Collectord instance run following command

Available from collectorfordocker v5.2

The following command takes several minutes.

docker exec collectorfordocker /collectord diag --stream 1>diag.tar.gz

You can extract a tar archive to verify the information that we collect. We include information about performance, memory usage, basic telemetry metrics, information file with the information of the host Linux version and basic information about the license.

Since 5.20.400 performance information is not collected by default unless you include a flag --include-performance-profiles in the command.

2. Collect logs

docker logs --details --timestamps collectorfordocker  1>collectorfordocker.log 2>&1

3. Run verify

Available from collectorfordocker v5.2

docker exec collectorfordocker /collectord verify > verify.log

4. Prepare tar archive

tar -czvf collectorfordocker-$(date +%s).tar.gz verify.log collectorfordocker.log diag.tar.gz

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.