Splunk Indexes

By default, collectorforopenshift forwards all the events to the default index specified for the HTTP Event Collector Token. Every HTTP Event Collector Token has a list of indexes where this specific Token can write data. One of the indexes from this list is also used as the default index when the sender of the data does not specify a target index. The application assumes that you are writing data to the indexes that are searchable by default by your Splunk Role. As an example, the main index is searchable by default.

If you used a different index that isn’t searchable by default by your Splunk Role, you would not see data on the dashboards.

To fix that, you can include this index in the Indexes searched by default for your role under Settings - Access Control

• Roles

Or you can change the Search Macros we use in the application and include a list of indexes you use for the Monitoring OpenShift events. You can find search macros in the Splunk Web UI under Settings - Advanced search

• Search macros (or by overriding $SPLUNK_HOME/etc/apps/monitoringopenshift/default/macros.conf with $SPLUNK_HOME/etc/apps/monitoringopenshift/local/macros.conf).

Starting from version 5.10, we include a base macro macro_openshift_base, where you can include the list of indexes only once; all other macros will use this configuration. For example

1macro_openshift_base = (index=openshift_stats OR index=openshift_logs)

If you want to have more precise configurations, you can modify specific macros.

1macro_openshift_stats = (index=openshift_stats sourcetype=openshift_stats)

You only need to update macros

• macro_openshift_events - all the OpenShift events.
• macro_openshift_host_logs - host logs.
• macro_openshift_logs - container logs.
• macro_openshift_proc_stats - proc metrics.
• macro_openshift_net_stats - network metrics.
• macro_openshift_net_socket_table - network socket tables.
• macro_openshift_mount_stats - container runtime storage usage metrics.
• macro_openshift_stats - system and container metrics.

Using dedicated indexes for different types of data

Considering the application access patterns and the content of the events, we recommend splitting logs from metrics and using dedicated indexes. For example, openshift_logs for events, container and host logs; openshift_stats for proc and system metrics; and openshift_prometheus for Prometheus metrics. You can also specify a dedicated index for every type of data the collectord forwards.

Using dedicated indexes also allows you to specify different retention policies for logs and metrics.

You can do that by using the Configuration Reference file and uncommenting the highlighted lines with the values of the indexes you want to use as the destination.

 1data:
 2  collector.conf: |
 3    ...
 4    
 5    [input.system_stats]
 6    
 7    ...
 8    
 9    # specify Splunk index
10    index = 
11    
12    ...
13    
14    [input.proc_stats]
15    
16    ...
17    
18    # specify Splunk index
19    index = 
20    
21    ...
22    
23    [input.net_stats]
24    
25    ...
26    
27    # specify Splunk index
28    index = 
29    
30    ...
31    
32    [input.net_socket_table]
33    
34    ...
35    
36    [input.mount_stats]
37    
38    ...
39    
40    # specify Splunk index
41    index = 
42    
43    ...
44    
45    # specify Splunk index
46    index = 
47    
48    ...
49    
50    [input.files]
51    
52    ...
53    
54    # specify Splunk index
55    index = 
56    
57    ...
58    
59    [input.files::syslog]
60    
61    ...
62    
63    # specify Splunk index
64    index =
65
66    ...
67    
68    [input.files::logs]
69    
70    ...
71    
72    # specify Splunk index
73    index =
74    
75    ...
76    
77    [input.kubernetes_events]
78    
79    ...
80    
81    # specify Splunk index
82    index =
83    
84    ...

Configuring dedicated indexes, source and sourcetype for Projects

You can also override targeted indexes for Projects in OpenShift. The collectord watches for annotations on the projects, workloads, and pods. For example, if you want to specify that you want to index all container logs, metrics, and events in the specific project project1 to index openshift_project1, you can annotate this project with

1oc annotate namespaces project1 \
2  collectord.io/index=openshift_project1

You can learn more about the available annotations at Annotations