Outcold Solutions LLC

Monitoring Kubernetes - Version 4

Forwarding Audit Logs

Our solution provides detailed Audit dashboards. By default, Kubernetes does not provide audit logs. You can enable them by following instructions from Kubernetes documentation Auditing .

You need to enable audit log only on Masters. For that, you need to edit definition of Kubernetes API Server. In case of clusters bootstrapped by kubeadm you can find the definition of Kubernetes API Server in file /etc/kubernetes/manifests/kube-apiserver.yaml. In other cases Kubernetes API Server Pod definition can be stored in /etc/kubernetes/manifests/apiserver.json.

Create Audit Policy file. Using example provided by the Kubernetes documentation save the file in /etc/kubernetes/policies/audit-policy.yaml.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # Do not log from very verbose system accounts
  - level: None
    users:
    - system:apiserver
    - system:kube-controller-manager
    - system:kube-scheduler

  # Do not log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: ""
      resources: ["endpoints", "services"]

  # Do not log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata

Configuration provided below set the policy file and tells to write logs directly to the standard output. Because Kubernetes API Server is running inside of the container, the collector forwards these logs automatically. We also need to mount audit policy file in the container that runs Kubernetes API Server. Modify /etc/kubernetes/manifests/kube-apiserver.yaml with the suggested changes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
...
spec:
  containers:
  - command:
    - kube-apiserver
...
    - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
    - --audit-log-path=-
    - --audit-log-format=json
...
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/kubernetes/policies
      name: policies
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/kubernetes/policies
      type: DirectoryOrCreate
    name: policies

To apply these changes you might need to restart kubelet.

sudo systemctl restart kubelet

Application has a macro, that defines how to find the audit logs macro_kubernetes_audit_logs.

(`macro_kubernetes_logs` OR `macro_kubernetes_host_logs`) "audit.k8s.io"

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which gives you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which helps developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.