Monitoring Docker. Configuring Splunk Indexes
collectorfordocker forwards all the events to the default index specified for HTTP Event Collector Token.
Every HTTP Event Collector Token has a list of indexes, where this specific Token can write data. One of the indexes
from this list is also used as a default index when the sender of the data does not specify target index.
The application assumes that you are writing data to the indexes, which are searchable by default by your Splunk Role.
As an example, the
main index is searchable by default.
If you used the different index, which isn't searchable by default by your Splunk Role, you would not see data on the dashboards.
To fix that, you can include this index to the Indexes searched by default for your role under Settings - Access Control - Roles
Or you can change Search Macros we use in the application and include a list of indexes you use for the Monitoring Docker
events. You can find search macros in Splunk Web UI under Settings - Advanced search
- Search macros (or by overriding
You need to modify macros definitions and add the indexes you use.
macro_docker_stats = (index=docker_stats sourcetype=docker_stats)
You need to update macros
macro_docker_events- all the docker events.
macro_docker_host_logs- host logs.
macro_docker_logs- container logs.
macro_docker_proc_stats- proc metrics.
macro_docker_stats- system and container metrics.
Using dedicated indexes for different types of data
Considering the application access patterns and the content of the events, we recommend to split logs with metrics
and use dedicated indexes. For example
docker_logs for events, container and host logs and
proc and system
metrics. You can also specify dedicated index for every type of the data collector forwards.
Using dedicated indexes allows you also to specify different retention policies for logs and the metrics.
You can do that by using Configuration Reference file and uncommenting highlighted lines with the values of the indexes you want to use as the destination.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57
... [input.system_stats] ... # specify Splunk index ; index = ... [input.proc_stats] ... # specify Splunk index ; index = ... [input.files] ... # specify Splunk index ; index = [input.files::syslog] ... # specify Splunk index ; index = ... [input.files::logs] ... # specify Splunk index ; index = ... [input.docker_events] ... # specify Splunk index ; index = ...