Outcold Solutions LLC

Monitoring Kubernetes - Version 5

Installation

With our solution for Monitoring Kubernetes, you can start monitoring your clusters in under 10 minutes, including forwarding metadata-enriched container logs, host logs, and metrics. Container image includes an evaluation license valid for the 30 days after the first start.

Splunk configuration

Install Monitoring Kubernetes application

Install latest version of application Monitoring Kubernetes from splunkbase. You need to install it on Search Heads only.

Enable HTTP Event Collector in Splunk

Outcold Solutions' Collector sends data to Splunk using HTTP Event Collector. By default, Splunk does not enable HTTP Event Collector. Please read HTTP Event Collector walkthrough to learn more about HTTP Event Collector.

The minimum requirement is Splunk Enterprise or Splunk Cloud 6.5. If you are managing Splunk Clusters with version below 6.5, please read our FAQ how to setup Heavy Weight Forwarder in between.

After enabling HTTP Event Collector, you need to find correct Url for HTTP Event Collector and generate an HTTP Event Collector Token. If you are running your Splunk instance on hostname hec.example.com, it listens on port 8088, using SSL and token is B5A79AAD-D822-46CC-80D1-819F80D7BFB0 you can test it with the curl command as in the example below.

$ curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}

-k is necessary for self-signed certificates.

If you use an index, that is not searchable by default, please read our documentation on how to configure indices at Splunk and inside the collector at Splunk Indexes.

Install Collector for Kubernetes

For Docker UCP installation see blog post Monitoring Docker Universal Control Plane (UCP) with Splunk Enterprise and Splunk Cloud

Pre-requirements

Collector works out of the box with CRI-O and Docker as runtime engines.

Our default configuration is optimized for the Kubernetes clusters deployed in Production environments, some data might now be available with minikube. For example minikube forwards host logs to journald without persistence on the disk and combines multiple control plane components into one process.

Docker Container Runtime

If you use Docker as a Container Runtime, the collector uses JSON-files generated by JSON logging driver as a source for container logs.

Some linux distributions, CentOS for example, by default enable journald logging driver instead of default JSON logging driver. You can verify which driver is used by default

$ docker info | grep "Logging Driver"
Logging Driver: json-file

If docker configuration file location is /etc/sysconfig/docker (common in CentOS/RHEL case with Docker 1.13), you can change it and restart docker daemon after that with following commands.

$ sed -i 's/--log-driver=journald/--log-driver=json-file --log-opt max-size=100M --log-opt max-file=3/' /etc/sysconfig/docker
$ systemctl restart docker

If you configure Docker daemon with daemon.json in /etc/docker/daemon.json (common in Debian/Ubuntu), you can change it and restart docker daemon.

{
  "log-driver": "json-file",
  "log-opts" : {
    "max-size" : "100m",
    "max-file" : "3"
  }
}
$ systemctl restart docker

Please follow the manual to learn how to configure default logging driver for containers:

JSON logging driver configuration

With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space. That is why we specify max-size and max-file with the default configurations. See Configure and troubleshoot the Docker daemon for more details.

Installation

Use latest Kubernetes configuration file collectorforkubernetes.yaml. This configuration deploys multiple workloads under collectorforkubernetes namespace.

Open it in your favorite editor and set the Splunk HTTP Event Collector Url, token, configuration for a certificate if required, review and accept a license agreement.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[general]

acceptEULA = false

...

# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url =

# Splunk HTTP Event Collector Token
token =

# Allow invalid SSL server certificate
insecure = false

# Path to CA certificate
caPath =

# CA Name to verify
caName =

Based on the example above you will need to modify the lines as in the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[general]

acceptEULA = true

...

# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url = https://hec.example.com:8088/services/collector/event/1.0

# Splunk HTTP Event Collector Token
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

# Allow invalid SSL server certificate
insecure = true

Apply this change to your Kubernetes cluster with kubectl

$ kubectl apply -f ./collectorforkubernetes.yaml

Verify the workloads.

$ kubectl get all --namespace collectorforkubernetes

Give it a few moments to download the image and start the containers. After all the pods are deployed, go to the Monitoring Kubernetes application in Splunk and you should see data on dashboards.

The collector forwards by default container logs, host logs (including syslog), metrics for host, pods, containers and processes.

Next steps

  • Review predefined alerts.
  • Verify configuration by using our troubleshooting instructions.
  • Enable Audit Logs. By default Kubernetes does not enable Audit Logs, if you want to be able to audit activities on Kubernetes API Server - you need to manually enable Audit Logs.
  • Verify Prometheus Metrics. Our configuration works in most of the times out of the box. If you will find that some of the data is not available for Control Plan, verify that you get all the Prometheus metrics and that all our configurations work in your cluster.
  • To learn how to forward application logs, please read our documentation on annotations.
  • We send the data to the default HTTP Event Collector index. For better performance we recommend at least to split logs with metrics in separate indices. You can find how to configure indexes in our guide Splunk Indices.
  • We provide flexible scheme, that allows you define search time extraction for logs in your containers. Follow the guide Splunk fields extraction for container logs to learn more.
  • You can define specific patterns for multi-line log lines; override indexes, sources, source types for the logs and metrics; extract fields, redirect some log lines to /dev/null, hide sensitive information from logs with annotations for pods.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.