Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Blog Contact Us Pricing Evaluation licenses Partners License Agreement Privacy Policy FAQ
Monitoring Kubernetes

Splunk field extraction

When you want Splunk to parse fields out of your container logs at search time, target the extraction at the right containers by matching on Collectord’s source value. Every container log Collectord forwards carries a structured source that encodes the container ID, container name, image name, pod name, namespace, and stream:

text
1/kubernetes/{kubernetes_container_id}/{kubernetes_container_name}/{kubernetes_image_name}/{kubernetes_pod_name}/{kubernetes_namespace}.{docker_stream}

Use that structure in props.conf with wildcards (*) for individual segments and ... to skip multiple — that way you can scope extractions to a single image, a specific container name, or any combination.

For example, to apply an nginx ingress controller access-log extraction to every container running gcr.io/google_containers/nginx-ingress-controller, regardless of container ID, container name, pod, namespace, or stream:

props.conf ini
1[source::/kubernetes/*/*/gcr.io/google_containers/nginx-ingress-controller:*/*/*]
2EXTRACT-nginx-ingress-controller-http = ^(?P&lt;remote_addr&gt;[^ ]+)\s+\-\s+\[(?P&lt;proxy_add_x_forwarded_for&gt;[^\]]+)\]\s+\-\s+(?P&lt;remote_user&gt;[^ ]+)\s+\[(?P&lt;time_local&gt;[^\]]+)[^"\n]*"(?P&lt;request&gt;[^"]+)"\s+(?P&lt;status&gt;\d+)\s+(?P&lt;body_bytes_sent&gt;\d+)\s+"(?P&lt;http_referer&gt;[^"]+)"\s+"(?P&lt;http_user_agent&gt;[^"]+)"\s+(?P&lt;request_length&gt;\d+)\s+(?P&lt;request_time&gt;[^ ]+)\s+\[(?P&lt;proxy_upstream_name&gt;[^\]]+)]\s+(?P&lt;upstream_addr&gt;[^\s]+)\s+(?P&lt;upstream_response_length&gt;\d+)\s+(?P&lt;upstream_response_time&gt;[^\s]+)\s+(?P&lt;upstream_status&gt;\d+)$</code></pre>

If you’d rather override the source or sourcetype on a per-pod or per-namespace basis instead — and key your extractions off those — see Splunk Indexes.