Monitoring Swarm Services with Monitoring Docker application

^{December 29, 2018}

Monitoring Docker application is a very generic application, that can help you to get started with various orchestration tools, which could be ECS or Docker Swarm or Docker UCP. We intentionally do not add Docker Swarm or ECS specific information in the Monitoring Docker application, as we do not want to overload this application with the orchestration tool data, that you don't use.

But the good news, if you want to have a nice dashboard that shows the overview of your service running on your Docker Swarm cluster, it is possible. This information is already getting collected with our Collectord. You need a few configuration changes in the application itself to make it possible to work with it.

At first we need to extract docker_stack_namespace, docker_service_name and docker_service_id, for that we can create a file $SPLUNK_ETC/apps/monitoringdocker/local/props.conf with following content

[docker_logs]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_stats]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_proc_stats]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_net_stats]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_net_socket_table]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

Alternatively, you can add these fields to the default configuration under $SPLUNK_ETC/apps/monitoringdocker/default/props.conf, make sure to keep it after the upgrades.

After that, you can leverage these fields and start querying information specific for some stack namespace or service name. As an example, we also provide a dashboard swarm.xml that you can import on your Splunk Search Heads to be able to overview whole service running on your Docker Swarm cluster.

Docker Swarm Services

Download dashboard.

About Outcold Solutions