Outcold Solutions LLC

Monitoring Windows Containers Installation

With our solution for Monitoring Windows Containers, you can start monitoring your clusters in under 10 minutes, including forwarding metadata-enriched container logs, host logs, and metrics.

Features:

  • Logs collection is based on native JSON logging driver.
  • Tiny image, tiny binary. Very low memory, CPU, and disk consumption.
  • Logs are enriched with Docker metadata (container, image, labels, etc.).
  • Collects container metrics, allowing you to correlate logs with metrics.
  • Collects process metrics.
  • Forwards docker service logs.
  • Uses HTTP Event Collector to ingest data in Splunk. Requires Splunk version 6.5 or above (talk to us if you need support for earlier version of Splunk).
  • Multi-line events support.
  • At least once delivery guarantee.

Splunk configuration

Install Monitoring Windows Containers

Install Monitoring Windows Containers from splunkbase. You need to install it on Search Heads only.

Enable HTTP Event Collector in Splunk

Outcold Solutions' Collector sends data to Splunk using HTTP Event Collector. By default, Splunk does not enable HTTP Event Collector. Please read HTTP Event Collector walkthrough to learn more about HTTP Event Collector.

After enabling HTTP Event Collector, you need to find correct Url for HTTP Event Collector and generate an HTTP Event Collector Token. If you are running your Splunk instance on hostname hec.example.com, it listens on port 8088, using SSL and token is B5A79AAD-D822-46CC-80D1-819F80D7BFB0 you can test it with the curl command as in the example below.

curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}

-k is necessary for self-signed certificates.

Install Collector for Windows Containers

Use latest available collector for Windows Containers distribution.

Open PowerShell as Administrator. Download collector, and extract the archive.

cd ~\Downloads
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri https://www.outcoldsolutions.com/docs/monitoring-wincontainers/monitoring-wincontainers.zip -OutFile monitoring-wincontainers.zip
Expand-Archive .\monitoring-wincontainers.zip -DestinationPath .\monitoring-wincontainers\

Create a folder for the collector.

mkdir C:\collector\

Copy collector.exe and collector.conf to this folder.

cp .\monitoring-wincontainers\collector\collector.exe c:\collector\
cp .\monitoring-wincontainers\collector\collector.conf c:\collector\

Create a folder for the collector database (where it stores metadata and positions of the files it reads).

mkdir C:\collector\data

Open collector.conf in your favorite editor to specify Splunk HTTP Event Collector URL and Token.

notepad C:\collector\collector.conf

In the opened file, find the stanza for the Splunk Output definition.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
; url =

# Splunk HTTP Event Collector Token
; token =

# Allow invalid SSL server certificate
; insecure = false

Uncomment all the required parameters and set the Splunk HTTP Event Collector URL, Token and insecure if required.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url = https://hec.example.com:8088/services/collector/event/1.0

# Splunk HTTP Event Collector Token
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

# Allow invalid SSL server certificate
insecure = true

If you use Splunk self-signed certificate, you want to add some SSL specific configuration. The easiest to get started is to set insecure = true to skip SSL validation, as we specified in the example above.

Important note, that collector does not require you to change the default logging driver. It implements collection on top of default JSON logging driver.

If you have installed docker not on C: drive, you need to find and update all default configurations where we reference disk C:, uncomment these values and change the drive to correct one.

Open a cmd.exe window or the PowerShell as the Administrator, go to the folder C:\collector and run command to verify that forwarding works as expected.

.\collector.exe monitor --environment=wincontainers --conf=C:\collector\collector.conf

You should see the output of the collector on the console, and in a few moments, you should start seeing data in Splunk. If you see that container works as expected, you can stop it and install it as the Windows Service.

New-Service -Name collectorforwincontainers -BinaryPathName 'C:\collector\collector.exe winservice --environment=wincontainers --conf=C:\collector\collector.conf' -DisplayName "Collector for Windows Containers" -StartupType 'Automatic' -DependsOn 'docker'

Press enter for requesting access as LocalSystem and after that start the service.

Start-Service collectorforwincontainers

Docker configuration

With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space. You can specify --log-driver=json-file --log-opt=max-size=100m --log-opt=max-file=5 for docker daemon. See Configure and troubleshoot the Docker daemon for more details.

Screencasts

Solution Overview

Release History

2.1 - 2018-01-15

Requires collectorfordocker version 2.1.66.180115 or above

  • Initial release with log forwarding, containers and processes metrics.

What is next?


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which gives you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which helps developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.