Monitoring Windows Containers Installation
With our solution for Monitoring Windows Containers, you can start monitoring your clusters in under 10 minutes, including forwarding metadata-enriched container logs, host logs, and metrics.
Features:
- Logs collection is based on native JSON logging driver.
- Tiny image, tiny binary. Very low memory, CPU, and disk consumption.
- Logs are enriched with Docker metadata (container, image, labels, etc.).
- Collects container metrics, allowing you to correlate logs with metrics.
- Collects process metrics.
- Forwards docker service logs.
- Uses HTTP Event Collector to ingest data in Splunk. Requires Splunk version 6.5 or above (talk to us if you need support for earlier version of Splunk).
- Multi-line events support.
- At least once delivery guarantee.
Splunk configuration
Install Monitoring Windows Containers
Install Monitoring Windows Containers from splunkbase. You need to install it on Search Heads only.
Enable HTTP Event Collector in Splunk
Outcold Solutions' Collector sends data to Splunk using HTTP Event Collector. By default, Splunk does not enable HTTP Event Collector. Please read HTTP Event Collector walkthrough to learn more about HTTP Event Collector.
After enabling HTTP Event Collector, you need to find correct Url for HTTP Event Collector and generate an HTTP Event Collector Token.
If you are running your Splunk instance on hostname hec.example.com,
it listens on port 8088, using SSL and token is B5A79AAD-D822-46CC-80D1-819F80D7BFB0
you can test it with the curl command as in the example below.
curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}' {"text": "Success", "code": 0}
-kis necessary for self-signed certificates.
Install Collector for Windows Containers
Use latest available collector for Windows Containers distribution.
Open PowerShell as Administrator. Download collector, and extract the archive.
cd ~\Downloads
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri https://www.outcoldsolutions.com/docs/monitoring-wincontainers/monitoring-wincontainers.zip -OutFile monitoring-wincontainers.zip
Expand-Archive .\monitoring-wincontainers.zip -DestinationPath .\monitoring-wincontainers\
Create a folder for the collector.
mkdir C:\collector\
Copy collector.exe and collector.conf to this folder.
cp .\monitoring-wincontainers\collector\collector.exe c:\collector\ cp .\monitoring-wincontainers\collector\collector.conf c:\collector\
Create a folder for the collector database (where it stores metadata and positions of the files it reads).
mkdir C:\collector\data
Open collector.conf in your favorite editor to specify Splunk HTTP Event Collector URL and Token.
notepad C:\collector\collector.conf
In the opened file, find the stanza for the Splunk Output definition.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | [general] ; license = # Splunk output [output.splunk] # Splunk HTTP Event Collector url ; url = # Splunk HTTP Event Collector Token ; token = # Allow invalid SSL server certificate ; insecure = false |
Uncomment all the required parameters and set the Splunk HTTP Event Collector URL, Token and insecure if required and include license key (request an evaluation license key with this automated form).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | [general] license = ... # Splunk output [output.splunk] # Splunk HTTP Event Collector url url = https://hec.example.com:8088/services/collector/event/1.0 # Splunk HTTP Event Collector Token token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0 # Allow invalid SSL server certificate insecure = true |
If you use Splunk self-signed certificate, you want to add some SSL specific configuration. The easiest to get started is to set
insecure = trueto skip SSL validation, as we specified in the example above.Important note, that collector does not require you to change the default logging driver. It implements collection on top of default JSON logging driver.
If you have installed docker not on
C:drive, you need to find and update all default configurations where we reference diskC:, uncomment these values and change the drive to correct one.
Open a cmd.exe window or the PowerShell as the Administrator, go to the folder C:\collector
and run command to verify that forwarding works as expected.
.\collector.exe monitor --environment=wincontainers --conf=C:\collector\collector.conf
You should see the output of the collector on the console, and in a few moments, you should start seeing data in Splunk. If you see that container works as expected, you can stop it and install it as the Windows Service.
New-Service -Name collectorforwincontainers -BinaryPathName 'C:\collector\collector.exe winservice --environment=wincontainers --conf=C:\collector\collector.conf' -DisplayName "Collector for Windows Containers" -StartupType 'Automatic' -DependsOn 'docker'
Press enter for requesting access as LocalSystem and after that start the service.
Start-Service collectorforwincontainers
Docker configuration
With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space.
You can specify
--log-driver=json-file --log-opt=max-size=100m --log-opt=max-file=5 for docker daemon.
See Configure and troubleshoot the Docker daemon for more details.
Screencasts
Solution Overview
Release History
2.1 - 2018-01-15
Requires collectorfordocker version 2.1.66.180115 or above
- Initial release with log forwarding, containers and processes metrics.
