Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Pricing Contact Us Blog Evaluation licenses Partners License Agreement Privacy Policy FAQ

Monitoring Kubernetes

Audit Logs

Our solution provides detailed Audit dashboards. By default, Kubernetes does not provide audit logs. You can enable them by following the instructions from the Kubernetes documentation Auditing.

You need to enable audit logs only on master nodes. For that, you need to edit the definition of the Kubernetes API Server. In the case of clusters bootstrapped by kubeadm, you can find the definition of the Kubernetes API Server in the file /etc/kubernetes/manifests/kube-apiserver.yaml. In other cases, the Kubernetes API Server Pod definition can be stored in /etc/kubernetes/manifests/apiserver.json.

Create an Audit Policy file. Use our example as a reference and save the file in /etc/kubernetes/policies/audit-policy.yaml.

Another good example of the audit-policy.yaml file is an audit profile used by GCE.

 1apiVersion: audit.k8s.io/v1beta1
 2kind: Policy
 3rules:
 4  # Do not log from kube-system accounts
 5  - level: None
 6    userGroups:
 7    - system:serviceaccounts:kube-system
 8  - level: None
 9    users:
10    - system:apiserver
11    - system:kube-scheduler
12    - system:volume-scheduler
13    - system:kube-controller-manager
14    - system:node
15
16  # Do not log from collector
17  - level: None
18    users:
19    - system:serviceaccount:collectorforkubernetes:collectorforkubernetes
20
21  # Don't log nodes communications
22  - level: None
23    userGroups:
24    - system:nodes
25
26  # Don't log these read-only URLs.
27  - level: None
28    nonResourceURLs:
29    - /healthz*
30    - /version
31    - /swagger*
32
33  # Log configmap and secret changes in all namespaces at the metadata level.
34  - level: Metadata
35    resources:
36    - resources: ["secrets", "configmaps"]
37
38  # A catch-all rule to log all other requests at the request level.
39  - level: Request

The configuration provided below sets the policy file and directs logs to be written directly to the standard output. Because the Kubernetes API Server is running inside the container, collectord forwards these logs automatically. We also need to mount the audit policy file in the container that runs the Kubernetes API Server. Modify /etc/kubernetes/manifests/kube-apiserver.yaml with the suggested changes.

 1...
 2spec:
 3  containers:
 4  - command:
 5    - kube-apiserver
 6...
 7    - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
 8    - --audit-log-path=-
 9    - --audit-log-format=json
10...
11    volumeMounts:
12    - mountPath: /etc/kubernetes/pki
13      name: k8s-certs
14      readOnly: true
15    - mountPath: /etc/ssl/certs
16      name: ca-certs
17      readOnly: true
18    - mountPath: /etc/kubernetes/policies
19      name: policies
20      readOnly: true
21  hostNetwork: true
22  volumes:
23  - hostPath:
24      path: /etc/kubernetes/pki
25      type: DirectoryOrCreate
26    name: k8s-certs
27  - hostPath:
28      path: /etc/ssl/certs
29      type: DirectoryOrCreate
30    name: ca-certs
31  - hostPath:
32      path: /etc/kubernetes/policies
33      type: DirectoryOrCreate
34    name: policies

To apply these changes you might need to restart kubelet.

1sudo systemctl restart kubelet

The application has a macro that defines how to find the audit logs: macro_kubernetes_audit_logs.

1(`macro_kubernetes_logs` OR `macro_kubernetes_host_logs`) "audit.k8s.io"

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.

Red Hat
Splunk
AWS