Monitoring Docker Universal Control Plane (UCP) with Splunk Enterprise and Splunk Cloud

[UPDATE (2018-11-14)] If you are using Docker UCP 3.1.0 or above please follow installation instructions from Installing Monitoring Kubernetes.

Docker Universal Control Plane (UCP) is the enterprise-grade cluster management solution from Docker. You install it on-premises or in your virtual private cloud, and it helps you manage your Docker cluster and applications through a single interface.

https://docs.docker.com/ee/ucp/

Docker UCP is the real king of orchestration, not only does it allow you to deploy workloads using docker-compose files, including docker services and docker stacks, but also it runs Kubernetes control plane and allows you to deploy Kubernetes Workloads.

It can be very challenging to set up infrastructure that will allow you to increase observability not only of your micro-services but also supporting infrastructure. Outcold Solutions offers dedicated solutions for Monitoring Docker and Monitoring Kubernetes, but if you are running UCP, which solution should you choose?

Both solutions will allow you to monitor all containers running on the cluster, including control plane containers, and application containers. If you deploy mostly Kubernetes workloads on UCP, you should consider using Monitoring Kubernetes solution. And if most of your applications are deployed with docker-compose files you should use Monitoring Docker, as Monitoring Kubernetes has additional concepts that do not apply to docker (Pods, Workloads).

Below we walk through how you can install both solutions, so you will be able to compare. In our scenarios, we used Docker EE with Universal Control Plane 3.0.5.

• Installing Monitoring Kubernetes on UCP
• Installing Monitoring Docker on UCP

For Docker UCP version 3.1.0 or above use Installing Monitoring Kubernetes instructions.

Installing Monitoring Kubernetes on UCP

A few details that you should be aware of regarding Kubernetes support on UCP:

• With UCP 3.0.5 it uses Kubernetes v1.8.11. In our example, we will use configuration built for Kubernetes 1.8.
• UCP does not use Kubernetes RBAC Authorization. It uses own User Management system. We will need to strip all RBAC related configuration from our manifest and configure service account with Docker UCP User Management.
• You cannot deploy DaemonSets on worker nodes outside of the kube-system namespace. For UCP deployment we change the namespace from collectorforkubernetes to kube-system.

The first step is simple, install our application from SplunkBase and enable HTTP Event Collector. Please follow our official guide on how to configure Splunk in Monitoring Kubernetes solution. As for collectord for kubernetes, the steps will be slightly different.

Grant collectorforkubernetes service account permissions to access Kubernetes API

First, you need to create a service account collectorforkubernetes using UCP. Go to the tab Service Accounts under Kubernetes and click the Create button.

Change namespace to kube-system and paste

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: collectorforkubernetes
  name: collectorforkubernetes
  namespace: kube-system

After creating this service account, we need to give it view-only permissions for Kubernetes API Service. You can do that with User Management by creating a new grant. Go to Grants under User Management and click the Create button.

In the wizard, on step 1, choose Service Account as a subject type, kube-system as a namespace, collectorforkubernetes as a Service Account and click Next

On step 2, choose View Only as a Role Type and click Next.

On step 3, choose namespaces as a Type and enable toggle Apply grant to all existing and new namespaces and click create.

Installing collectorforkubernetes

Download collectorforkubernetes.yaml that we specifically prepared for UCP deployment.

Similarly to the general installation instructions you need to accept the License, configure Splunk URL and Token and include license key (request an evaluation license key with this automated form).

[general]

acceptLicense = true

license = ...

...

# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url = https://hec.example.com:8088/services/collector/event/1.0

# Splunk HTTP Event Collector Token
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

# Allow invalid SSL server certificate
insecure = true

Copy the whole content of the YAML file, and go to the UCP console, go to Controllers under Kubernetes, and click the Create button.

Change namespace to collectorforkubernetes and paste the whole content to the Object YAML section and click Create.

If everything is correct, you should start seeing data in a few moments in the Monitoring Kubernetes application in Splunk.

Within the application, when you navigate to a specific node, you will be able to see pods scheduled with Kubernetes

And below you will be able to see all containers that have been scheduled with Kubernetes or Docker Services and Stacks

Please read Next Steps that we recommend after installation.

Installing Monitoring Docker on UCP

At first install our application from SplunkBase and enable HTTP Event Collector. Please follow our official guide on how to configure Splunk in Monitoring Docker solution.

To install collectord on your Docker nodes we recommend using the CLI, as our configuration has a lot of mounts and it is easy to make a mistake by adding them manually.

To get access to the CLI from UCP, you can find instructions on the main Dashboard if you scroll to the very bottom of the page

After configuring the CLI, create a file collectorfordocker.yaml with the content as in the example below. Specify the correct Splunk URL and Token and accept the License.

version: "3"
services:

  collectorfordocker:
    image: outcoldsolutions/collectorfordocker:5.2
    volumes:
      - /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro
      - /proc:/rootfs/proc:ro
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/:/rootfs/var/lib/docker/:ro
      - /var/run/docker.sock:/rootfs/var/run/docker.sock:ro
      - collector_data:/data/
    environment:
      - COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0
      - COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0
      - COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true
      - COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true
      - COLLECTOR__LICENSE=general__license=...
      - COLLECTOR__CGROUPS=general.docker__containersCgroupFilter=^(/([^/\s]+/)*(docker-|docker/|kubepods/.*)[0-9a-f]{64}(\.scope)?)$$
    deploy:
      mode: global
      restart_policy:
        condition: any
      resources:
        limits:
          cpus: '1'
          memory: 256M
        reservations:
          cpus: '0.1'
          memory: 64M

volumes:
  collector_data:

Create services with the Docker CLI

docker stack deploy --compose-file ./collectorfordocker.yml collectorfordocker

Check that services have been deployed

docker stack services collectorfordocker

Give it a few moments, and you should see the data in the Monitoring Docker application.

Similarly to Monitoring Kubernetes application, you will be able to see all containers running on your Docker UCP cluster

Please read Next Steps that we recommend after installation.

Summary

Both applications, Monitoring Docker and Monitoring Kubernetes, provide you with a way to monitor your clusters, see logs from the containers and from the hosts as well. Monitoring Kubernetes also provides dashboards dedicated for the Kubernetes Control Plane.

If you prefer to use both applications, it is possible to add aliases for Monitoring Docker application to reuse the data that we forward for the Monitoring Kubernetes application.

Have a question? We are one email away.