Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Swarm Services with Monitoring Docker application

Monitoring Docker application is a very generic application that can help you to get started with various orchestration tools, which could be ECS or Docker Swarm or Docker UCP. We intentionally do not add Docker Swarm or ECS-specific information in the Monitoring Docker application, as we do not want to overload this application with the orchestration tool data that you don’t use.

But the good news is that if you want to have a nice dashboard that shows the overview of your service running on your Docker Swarm cluster, it is possible. This information is already being collected by our Collectord. You need a few configuration changes in the application itself to make this possible.

First, we need to extract docker_stack_namespace, docker_service_name and docker_service_id. For that, we can create a file $SPLUNK_ETC/apps/monitoringdocker/local/props.conf with the following content

[docker_logs]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_stats]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_proc_stats]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_net_stats]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

[docker_net_socket_table]
EVAL-docker_stack_namespace = substr(mvfilter(match(docker_container_labels, "^com\.docker\.stack\.namespace=")), 28)
EVAL-docker_service_name = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.name=")), 31)
EVAL-docker_service_id = substr(mvfilter(match(docker_container_labels, "^com\.docker\.swarm\.service\.id=")), 29)

Alternatively, you can add these fields to the default configuration under $SPLUNK_ETC/apps/monitoringdocker/default/props.conf, make sure to keep it after upgrades.

After that, you can leverage these fields and start querying information specific to some stack namespace or service name. As an example, we also provide a dashboard swarm.xml that you can import to your Splunk Search Heads to be able to overview the whole service running on your Docker Swarm cluster.

Docker Swarm Services

Download dashboard.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.