Splunk fields extraction for container logs

For container logs forwarded by collectord, it is possible to specify field extraction rules specific to image names, container names, or a combination of them.

All container logs have a source format that includes the container ID, container name, image name, pod name, namespace, and stream.

/kubernetes/{kubernetes_container_id}/{kubernetes_container_name}/{kubernetes_image_name}/{kubernetes_pod_name}/{kubernetes_namespace}.{docker_stream}

Using this knowledge, you can create field extraction rules for a specific image or container, including glob patterns using wildcards and ... for skipping multiple parts of the path.

As an example, you can specify field extraction for an nginx container in props.conf using wildcard characters for the container ID, container name, and docker stream. This field extraction applies to all containers created from the gcr.io/google_containers/nginx-ingress-controller image.

[source::/kubernetes/*/*/gcr.io/google_containers/nginx-ingress-controller:*/*/*]
EXTRACT-nginx-ingress-controller-http = ^(?P&lt;remote_addr&gt;[^ ]+)\s+\-\s+\[(?P&lt;proxy_add_x_forwarded_for&gt;[^\]]+)\]\s+\-\s+(?P&lt;remote_user&gt;[^ ]+)\s+\[(?P&lt;time_local&gt;[^\]]+)[^"\n]*"(?P&lt;request&gt;[^"]+)"\s+(?P&lt;status&gt;\d+)\s+(?P&lt;body_bytes_sent&gt;\d+)\s+"(?P&lt;http_referer&gt;[^"]+)"\s+"(?P&lt;http_user_agent&gt;[^"]+)"\s+(?P&lt;request_length&gt;\d+)\s+(?P&lt;request_time&gt;[^ ]+)\s+\[(?P&lt;proxy_upstream_name&gt;[^\]]+)]\s+(?P&lt;upstream_addr&gt;[^\s]+)\s+(?P&lt;upstream_response_length&gt;\d+)\s+(?P&lt;upstream_response_time&gt;[^\s]+)\s+(?P&lt;upstream_status&gt;\d+)$</code></pre>

You can also override source and source type with annotations. See Splunk Indexes.

Links

• Installation
  • Start monitoring your Kubernetes environments in under 10 minutes.
  • Automatically forward host, container and application logs.
  • Test our solution with the embedded 30-day evaluation license.
• Collectord Configuration
  • Collectord configuration reference.
• Annotations
  • Changing index, source, sourcetype for namespaces, workloads and pods.
  • Forwarding application logs.
  • Multi-line container logs.
  • Fields extraction for application and container logs (including timestamp extractions).
  • Hiding sensitive data, stripping terminal escape codes and colors.
  • Forwarding Prometheus metrics from Pods.
• Audit Logs
  • Configure audit logs.
  • Forwarding audit logs.
• Prometheus metrics
  • Collect metrics from control plane (etcd cluster, API server, kubelet, scheduler, controller).
  • Configure the collectord to forward metrics from the services in Prometheus format.
• Configuring Splunk Indexes
  • Using non-default HTTP Event Collector index.
  • Configure the Splunk application to use indexes that are not searchable by default.
• Splunk fields extraction for container logs
  • Configure search-time field extractions for container logs.
  • Container logs source pattern.
• Configurations for Splunk HTTP Event Collector
  • Configure multiple HTTP Event Collector endpoints for Load Balancing and Fail-overs.
  • Secure HTTP Event Collector endpoint.
  • Configure the Proxy for HTTP Event Collector endpoint.
• Monitoring multiple clusters
  • Learn how to monitor multiple clusters.
  • Learn how to set up ACL in Splunk.
• Streaming Kubernetes Objects from the API Server
  • Learn how to stream all changes from the Kubernetes API Server.
  • Stream changes and objects from Kubernetes API Server, including Pods, Deployments or ConfigMaps.
• License Server
  • Learn how to configure a remote License URL for Collectord.
• Monitoring GPU
• Alerts
• Troubleshooting
• Release History
• Upgrade instructions
• Security
• FAQ and the common questions
• License agreement
• Pricing
• Contact