Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Windows Containers

Monitoring Windows Containers Configuration

Collectord configuration

Join Rules

By default collectord joins all messages with previous if they start with spaces. Below you can find how to specify a custom rule on the example of java application.

If this is a sample of the application logs.

[2017-09-04T06:28:05,664][WARN ][MyComponent]
java.security.AccessControlException: access denied
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
  at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
[2017-09-04T06:28:05,664][WARN ][MyComponent] another message

You can specify the join rules, where you configure that you want to match all containers with the name that contains my_app in their name, and pattern for the new message should match regex ^\[\d{4}-.

[pipe.join::my_app]
matchRegex.docker_container_name = .+my_app.+
patternRegex = ^\[\d{4}-

collector.conf Reference

# collectord configuration file
#
# Run collectord with flag -conf and specify location of the configuration file.
#
# You can override all the values using environment variables with the format like
#   COLLECTOR__<section>__<key>=<value>
# As an example you can set dataPath in [general] section as
#   COLLECTOR__general__dataPath=C:\\some\\path\\data.db
# This parameter can be configured using -env-override, set it to empty string to disable this feature

[general]

# location for the database
# is used to store position of the files and internal state
; dataPath = .\data\

# log level (trace, debug, info, warn, error, fatal)
; logLevel = info

# http server gives access to two endpoints
# /healthz
# /metrics
; httpServerBinding = :8080

# telemetry report endpoint, set it to empty string to disable telemetry
; telemetryEndpoint = https://license.outcold.solutions/telemetry/

# license check endpoint
; licenseEndpoint = https://license.outcold.solutions/license/

# license server through proxy
; licenseServerProxyUrl =

# license
; license =

# docker daemon hostname is used by default as hostname
# use this configuration to override
; hostname =


[general.docker]

# path to docker root folder
; dockerRootFolder = C:\ProgramData\Docker\


# Log files
[input.files]

# disable container logs monitoring
; disabled = false

# root location of docker files
; path = C:\ProgramData\Docker\containers\

# glob matching pattern for log files
; glob = *\*-json.log*

# files are read using polling schema, when reach the EOF how often to check if files got updated
; pollingInterval = 250ms

# how often to look for the new files under logs path
; walkingInterval = 5s

# include verbose fields in events (file offset)
; verboseFields = false

# override type
; type = wincontainers_logs

# read hardlinks instead of original files
# on Windows that allows not to block the original directory, when
# files are getting deleted by daemon
; hardlinks = true
; hardlinksPath = C:\ProgramData\Docker\containers\


[input.files::docker_service]

# disable host level logs
; disabled = false

# root location of docker files
path = C:\ProgramData\Docker\

# regex matching pattern
match = ^service.txt$

# limit search only on one level
recursive = false

# files are read using polling schema, when reach the EOF how often to check if files got updated
pollingInterval = 250ms

# how often o look for the new files under logs path
walkingInterval = 5s

# include verbose fields in events (file offset)
verboseFields = false

# override type
type = wincontainers_host_logs

# field extraction
extraction = ^\[(?P<timestamp>[^\]]+)\]\[(?P<docker_service_component>[^\]]+)\]\[(?P<docker_service_level>[^\]]+)\]\s(.*)$
# extractionMessageField =

# timestamp field
timestampField = timestamp

# format for timestamp
# the layout defines the format by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006`
timestampFormat = 15:04:05.000

# Adjust date, if month/day aren't set in format
timestampSetMonth = true
timestampSetDay = true

# timestamp location (if not defined by format)
timestampLocation = Local


# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
; url =

# Splunk HTTP Event Collector Token
; token =

# Allow invalid SSL server certificate
; insecure = false

# Path to CA cerificate
; caPath =

# CA Name to verify
; caName =

# Events are batched with the maximum size set by batchSize and staying in pipeline for not longer
# than set by frequency
; frequency = 5s
; batchSize = 768K

# Splunk through proxy
; proxyUrl =


# Pipe to join events (container logs only)
[pipe.join]

# disable joining event
; disabled = false

# Maximum interval of messages in pipeline
; maxInterval = 100ms

# Maximum time to wait for the messages in pipeline
; maxWait = 1s

# Maximum message size
; maxSize = 100K

# Default pattern to indicate new message (should start not from space)
; patternRegex = ^[^\s]


# Define special event join patterns for matched events
# Section consist of [pipe.join::<name>]
# [pipe.join::my_app]
## Set match pattern for the fields
#; matchRegex.docker_container_image = my_app
#; matchRegex.docker_stream = stdout
## All events start from '[<digits>'
#; patternRegex = ^\[\d+


# Stats for Windows Containers
[input.wincontainer_stats]

; disabled = false

# how often to collect stats
; statsInterval = 30s

# override type
; type = wincontainers_stats


# Not supported on Windows
[input.docker_events]
; disabled = true


# Not supported on Windows
[input.system_stats]
; disabled = true


# Not supported on Windows
[input.proc_stats]
; disabled = true

What is next?


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.