Outcold Solutions LLC

Monitoring Windows Containers Configuration

Collector configuration

Join Rules

By default collector joins all messages with previous if they start with spaces. Below you can find how to specify a custom rule on the example of java application.

If this is a sample of the application logs.

[2017-09-04T06:28:05,664][WARN ][MyComponent]
java.security.AccessControlException: access denied
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
  at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
[2017-09-04T06:28:05,664][WARN ][MyComponent] another message

You can specify the join rules, where you configure that you want to match all containers with the name that contains my_app in their name, and pattern for the new message should match regex ^\[\d{4}-.

[pipe.join::my_app]
matchRegex.docker_container_name = .+my_app.+
patternRegex = ^\[\d{4}-

collector.conf Reference

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
# collector configuration file
#
# Run collector with flag -conf and specify location of the configuration file.
#
# You can override all the values using environment variables with the format like
#   COLLECTOR__<section>__<key>=<value>
# As an example you can set dataPath in [general] section as
#   COLLECTOR__general__dataPath=C:\\some\\path\\data.db
# This parameter can be configured using -env-override, set it to empty string to disable this feature

[general]

# location for the database
# is used to store position of the files and internal state
; dataPath = .\data\

# log level (trace, debug, info, warn, error, fatal)
; logLevel = info

# http server gives access to two endpoints
# /healthz
# /metrics
; httpServerBinding = :8080

# telemetry report endpoint, set it to empty string to disable telemetry
; telemetryEndpoint = https://license.outcold.solutions/telemetry/

# license check endpoint
; licenseEndpoint = https://license.outcold.solutions/license/

# license server through proxy
; licenseServerProxyUrl =

# license
; license =

# docker daemon hostname is used by default as hostname
# use this configuration to override
; hostname =


[general.docker]

# path to docker root folder
; dockerRootFolder = C:\ProgramData\Docker\


# Log files
[input.files]

# disable container logs monitoring
; disabled = false

# root location of docker files
; path = C:\ProgramData\Docker\containers\

# glob matching pattern for log files
; glob = *\*-json.log*

# files are read using polling schema, when reach the EOF how often to check if files got updated
; pollingInterval = 250ms

# how often to look for the new files under logs path
; walkingInterval = 5s

# include verbose fields in events (file offset)
; verboseFields = false

# override type
; type = wincontainers_logs

# read hardlinks instead of original files
# on Windows that allows not to block the original directory, when
# files are getting deleted by daemon
; hardlinks = true
; hardlinksPath = C:\ProgramData\Docker\containers\


[input.files::docker_service]

# disable host level logs
; disabled = false

# root location of docker files
path = C:\ProgramData\Docker\

# regex matching pattern
match = ^service.txt$

# limit search only on one level
recursive = false

# files are read using polling schema, when reach the EOF how often to check if files got updated
pollingInterval = 250ms

# how often o look for the new files under logs path
walkingInterval = 5s

# include verbose fields in events (file offset)
verboseFields = false

# override type
type = wincontainers_host_logs

# field extraction
extraction = ^\[(?P<timestamp>[^\]]+)\]\[(?P<docker_service_component>[^\]]+)\]\[(?P<docker_service_level>[^\]]+)\]\s(.*)$

# timestamp field
timestampField = timestamp

# format for timestamp
# the layout defines the format by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006`
timestampFormat = 15:04:05.000

# Adjust date, if month/day aren't set in format
timestampSetMonth = true
timestampSetDay = true

# timestamp location (if not defined by format)
timestampLocation = Local


# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
; url =

# Splunk HTTP Event Collector Token
; token =

# Allow invalid SSL server certificate
; insecure = false

# Path to CA cerificate
; caPath =

# CA Name to verify
; caName =

# Events are batched with the maximum size set by batchSize and staying in pipeline for not longer
# than set by frequency
; frequency = 5s
; batchSize = 768K

# Splunk through proxy
; proxyUrl =


# Pipe to join events (container logs only)
[pipe.join]

# disable joining event
; disabled = false

# Maximum interval of messages in pipeline
; maxInterval = 100ms

# Maximum time to wait for the messages in pipeline
; maxWait = 1s

# Maximum message size
; maxSize = 100K

# Default pattern to indicate new message (should start not from space)
; patternRegex = ^[^\s]


# Define special event join patterns for matched events
# Section consist of [pipe.join::<name>]
# [pipe.join::my_app]
## Set match pattern for the fields
#; matchRegex.docker_container_image = my_app
#; matchRegex.docker_stream = stdout
## All events start from '[<digits>'
#; patternRegex = ^\[\d+


# Stats for Windows Containers
[input.wincontainer_stats]

; disabled = false

# how often to collect stats
; statsInterval = 30s

# override type
; type = wincontainers_stats


# Not supported on Windows
[input.docker_events]
; disabled = true


# Not supported on Windows
[input.system_stats]
; disabled = true


# Not supported on Windows
[input.proc_stats]
; disabled = true

What is next?


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.