Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Blog Contact Us Pricing Evaluation licenses Partners License Agreement Privacy Policy FAQ
Monitoring Windows Containers

Installation

This guide walks you through installing Monitoring Windows Containers end-to-end: configuring the Splunk app and HTTP Event Collector, then deploying Collectord on your Windows hosts to forward metadata-enriched container logs, host logs, and metrics. A typical install takes under 10 minutes. If you don’t have a license yet, you can request a 30-day evaluation.

Features:

  • Log collection on top of the native JSON logging driver — no driver swap required.
  • Tiny image, tiny binary. Low memory, CPU, and disk footprint.
  • Logs enriched with Docker metadata (container, image, labels).
  • Container metrics alongside logs, so you can correlate the two.
  • Process metrics.
  • Docker service log forwarding.
  • HTTP Event Collector for ingest into Splunk. Requires Splunk 6.5 or above — talk to us if you need support for an earlier version.
  • Multi-line event support.
  • At-least-once delivery guarantee.

Splunk configuration

Install the Monitoring Windows Containers application

Install Monitoring Windows Containers from Splunkbase on your Search Heads only.

Enable HTTP Event Collector in Splunk

Collectord forwards data to Splunk over the HTTP Event Collector (HEC). HEC is off by default — read the HTTP Event Collector walkthrough to enable it.

Once HEC is enabled, you need two pieces of information for the rest of this guide: the HEC endpoint URL and an HEC token. If your Splunk instance is on hec.example.com listening on port 8088 over SSL, with a token of B5A79AAD-D822-46CC-80D1-819F80D7BFB0, you can verify both with curl:

bash
1curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
2{"text": "Success", "code": 0}

-k skips certificate validation; use it only for self-signed certificates.

Install Collectord for Windows Containers

Use the latest available Collectord for Windows Containers distribution.

Open PowerShell as Administrator, then download Collectord and extract the archive:

batch
1cd ~\Downloads
2[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
3Invoke-WebRequest -Uri https://www.outcoldsolutions.com/docs/monitoring-wincontainers/monitoring-wincontainers.zip -OutFile monitoring-wincontainers.zip
4Expand-Archive .\monitoring-wincontainers.zip -DestinationPath .\monitoring-wincontainers\

Create a folder for Collectord:

batch
1mkdir C:\collector\

Copy collector.exe and collector.conf into it:

batch
1cp .\monitoring-wincontainers\collector\collector.exe c:\collector\
2cp .\monitoring-wincontainers\collector\collector.conf c:\collector\

Create a folder for Collectord’s database — it stores file metadata and read positions there:

batch
1mkdir C:\collector\data

Open collector.conf in your editor of choice to set the Splunk HEC URL and token:

batch
1notepad C:\collector\collector.conf

Find the Splunk output stanza:

collector.conf ini
 1[general]
 2
 3; license =
 4
 5# Splunk output
 6[output.splunk]
 7
 8# Splunk HTTP Event Collector url
 9; url =
10
11# Splunk HTTP Event Collector Token
12; token =
13
14# Allow invalid SSL server certificate
15; insecure = false

Uncomment the required parameters, fill in the HEC URL, token, and insecure if needed, and paste in your license key (request an evaluation key with this automated form):

collector.conf ini
 1[general]
 2
 3license = ...
 4
 5# Splunk output
 6[output.splunk]
 7
 8# Splunk HTTP Event Collector url
 9url = https://hec.example.com:8088/services/collector/event/1.0
10
11# Splunk HTTP Event Collector Token
12token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
13
14# Allow invalid SSL server certificate
15insecure = true

If Splunk uses a self-signed certificate, you’ll want a few SSL-specific settings. The quickest way to get going is insecure = true to skip SSL validation, as shown above.

Collectord doesn’t require you to change Docker’s default logging driver — it reads from the default JSON logging driver in place.

If Docker is installed somewhere other than the C: drive, find every default config that references C:, uncomment those values, and point them at the correct drive.

Open cmd.exe or PowerShell as Administrator, switch to C:\collector, and run Collectord in the foreground to verify forwarding works:

batch
1.\collector.exe monitor --environment=wincontainers --conf=C:\collector\collector.conf

You should see Collectord output on the console and, within a few moments, data flowing into Splunk. Once that looks right, stop it and install it as a Windows Service:

batch
1New-Service -Name collectorforwincontainers -BinaryPathName 'C:\collector\collector.exe winservice --environment=wincontainers --conf=C:\collector\collector.conf' -DisplayName "Collector for Windows Containers" -StartupType 'Automatic' -DependsOn 'docker'

Press enter to accept LocalSystem for the service account, then start the service:

batch
1Start-Service collectorforwincontainers

Docker configuration

By default, Docker doesn’t rotate JSON log files — they grow until they fill the disk. Pass --log-driver=json-file --log-opt=max-size=100m --log-opt=max-file=5 to the Docker daemon to cap them. See Configure and troubleshoot the Docker daemon for details.

Screencasts

Solution Overview

Release History

2.1 - 2018-01-15

Requires collectorfordocker version 2.1.66.180115 or above

  • Initial release with log forwarding, containers and processes metrics.

What is next?