Outcold Solutions LLC

Monitoring Docker Universal Control Plane (UCP) with Splunk Enterprise and Splunk Cloud

October 17, 2018

[UPDATE (2018-11-14)] If you are using Docker UCP 3.1.0 or above please follow installation instructions from Installing Monitoring Kubernetes.


Docker Universal Control Plane (UCP) is the enterprise-grade cluster management solution from Docker. You install it on-premises or in your virtual private cloud, and it helps you manage your Docker cluster and applications through a single interface.

https://docs.docker.com/ee/ucp/

Docker UCP is a real king of orchestration, not only it allows you to deploy workloads using docker-compose files, including docker services and docker stacks, but also it runs Kubernetes control plane and allows you to deploy Kubernetes Workloads.

It can be very challenging to set up infrastructure that will have you to increase observability not only of your micro-services but also supporting infrastructure. Outcold Solutions offers dedicated solutions for Monitoring Docker and Monitoring Kubernetes, but if you are running UCP, which solution should you choose?

Both solutions will allow you to monitor all containers running on the cluster, including control plane containers, and application containers. If you deploy mostly Kubernetes workloads on UCP, you should consider using Monitoring Kubernetes solution. And if most of your applications are deployed with docker-compose files you should use Monitoring Docker, as Monitoring Kubernetes has additional concepts that do not apply to docker (Pods, Workloads).

Below we walk through how you can install both solutions, so you will be able to compare. In our scenarios, we used Docker EE with Universal Control Plane 3.0.5.

For Docker UCP version 3.1.0 or above use Installing Monitoring Kubernetes instructions.

Installing Monitoring Kubernetes on UCP

Few details that you should be aware of Kubernetes support on UCP:

  • With UCP 3.0.5 it uses Kubernetes v1.8.11. In our example, we will use configuration built for Kubernetes 1.8.
  • UCP does not use Kubernetes RBAC Authorization. It uses own User Management system. We will need to strip all RBAC related configuration from our manifest and configure service account with Docker UCP User Management.
  • You cannot deploy DaemonSets on Worker nodes outside of the kube-system namespace. For UCP deployment we change the namespace from collectorforkubernetes to kube-system.

The first step is simple, install our application from SplunkBase and enable HTTP Event Collector. Please follow our official guide on how to configure Splunk in Monitoring Kubernetes solution. As for collector for kubernetes, the steps will be slightly different.

Grant collectorforkubernetes service account permissions to access Kubernetes API

At first you need to create service account collectorforkubernetes using UCP. Go to the tab Service Accounts under Kubernetes and click a Create button.

UCP service account

Change namespace to kube-system and paste

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: collectorforkubernetes
  name: collectorforkubernetes
  namespace: kube-system

Create service account

After creating this service account we need to give it View only permissions for Kubernetes API Service. You can do that with User Management, by creating a new grant. Go to the Grants under User Management and click on Create button.

UCP user management

In the wizard on step 1 choose Service Account as a subject type, kube-system as a namespace, collectorforkubernetes as a Service Account and click Next

UCP grant - step 1

On step 2 choose View Only as a Role Type and click Next.

UCP grant - step 2

On step 3 choose namespaces as a Type and enable toggle Apply grant to all existing and new namespaces and click create.

UCP grant - step 3

Installing collectorforkubernetes

Download collectorforkubernetes.yaml, that we specifically prepared for UCP deployment.

Similarly to the general installations instructions you need to accept License, configure Splunk URL and Token and include license key (request an evaluation license key with this automated form).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
[general]

acceptLicense = true

license = ...

...

# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url = https://hec.example.com:8088/services/collector/event/1.0

# Splunk HTTP Event Collector Token
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

# Allow invalid SSL server certificate
insecure = true

Copy whole content of YAML file, and go to the UCP console, Controllers under Kubernetes and click Create button.

UCP controllers create

Change namespace to collectorforkubernetes and paste whole content to the Object YAML section and click Create.

UCP controllers create

If everything is correct, you should start seeing data in a few moments in the Monitoring Kubernetes application in Splunk.

Monitoring Kubernetes - UCP

Within the application when you navigate to a specific node, you will be able to see pods scheduled with kubernetes

List Pods

And below you will be able to see all containers, that have been scheduled with Kubernetes or Docker Services and Stacks

List Containers

Please read Next Steps that we recommend after installation.

Installing Monitoring Docker on UCP

At first install our application from SplunkBase and enable HTTP Event Collector. Please follow our official guide on how to configure Splunk in Monitoring Docker solution.

To install collector on your Docker nodes we recommend to use CLI, as our configuration has a lot of mounts, and it is easy to make a mistake by adding them manually.

To get access to the CLI from UCP, you can find instructions on the main Dashboard, if you will scroll to the very bottom of the page

List Containers

After configuring CLI, create a file collectorfordocker.yaml with the content as in the example below. Specify correct Splunk URL and Token and accept License.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
version: "3"
services:

  collectorfordocker:
    image: outcoldsolutions/collectorfordocker:5.2
    volumes:
      - /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro
      - /proc:/rootfs/proc:ro
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/:/rootfs/var/lib/docker/:ro
      - /var/run/docker.sock:/rootfs/var/run/docker.sock:ro
      - collector_data:/data/
    environment:
      - COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0
      - COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0
      - COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true
      - COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true
      - COLLECTOR__LICENSE=general__license=...
      - COLLECTOR__CGROUPS=general.docker__containersCgroupFilter=^(/([^/\s]+/)*(docker-|docker/|kubepods/.*)[0-9a-f]{64}(\.scope)?)$$
    deploy:
      mode: global
      restart_policy:
        condition: any
      resources:
        limits:
          cpus: '1'
          memory: 256M
        reservations:
          cpus: '0.1'
          memory: 64M

volumes:
  collector_data:

Create services with docker CLI

docker stack deploy --compose-file ./collectorfordocker.yml collectorfordocker

Check that services have been deployed

docker stack services collectorfordocker

Give it a few moments, and you should see the data in the Monitoring Docker application.

Monitoring Docker

Similarly to Monitoring Kubernetes application, you will be able to see all containers running on your Docker UCP cluster

Monitoring Docker - Containers

Please read Next Steps that we recommend after installation.

Summary

Both application Monitoring Docker and Monitoring Kubernetes provides you a way to monitor your clusters, see logs from the containers and from the hosts as well. Monitoring Kubernetes also provides dashboards dedicated for the Kubernetes Control Plane.

If you prefer to use both applications, it is possible to add an aliases for Monitoring Docker application to reuse the data, that we forward for Monitoring Kubernetes application.

Have a question? We are one email away.

docker, kubernetes, ucp, splunk, splunk enterprise, splunk cloud, monitoring docker, log forwarding

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.