Forwarding Audit Logs

Kubernetes

Our solution provides detailed Audit dashboards. By default, Kubernetes does not provide audit logs. You can enable them by following instructions from Kubernetes documentation Auditing .

You need to enable audit log only on Masters. For that, you need to edit definition of Kubernetes API Server. In case of clusters bootstrapped by kubeadm you can find the definition of Kubernetes API Server in file /etc/kubernetes/manifests/kube-apiserver.yaml. In other cases Kubernetes API Server Pod definition can be stored in /etc/kubernetes/manifests/apiserver.json.

Create Audit Policy file. Use our example as a reference and save the file in /etc/kubernetes/policies/audit-policy.yaml.

Another good example of the audit-policy.yaml file is an audit profile used by GCE.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # Do not log from kube-system accounts
  - level: None
    userGroups:
    - system:serviceaccounts:kube-system
  - level: None
    users:
    - system:apiserver
    - system:kube-scheduler
    - system:volume-scheduler
    - system:kube-controller-manager
    - system:node

  # Do not log from collector
  - level: None
    users:
    - system:serviceaccount:collectorforkubernetes:collectorforkubernetes

  # Don't log nodes communications
  - level: None
    userGroups:
    - system:nodes

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
    - /healthz*
    - /version
    - /swagger*

  # Log configmap and secret changes in all namespaces at the metadata level.
  - level: Metadata
    resources:
    - resources: ["secrets", "configmaps"]

  # A catch-all rule to log all other requests at the request level.
  - level: Request

Configuration provided below set the policy file and tells to write logs directly to the standard output. Because Kubernetes API Server is running inside of the container, the collector forwards these logs automatically. We also need to mount audit policy file in the container that runs Kubernetes API Server. Modify /etc/kubernetes/manifests/kube-apiserver.yaml with the suggested changes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

...
spec:
  containers:
  - command:
    - kube-apiserver
...
    - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
    - --audit-log-path=-
    - --audit-log-format=json
...
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/kubernetes/policies
      name: policies
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/kubernetes/policies
      type: DirectoryOrCreate
    name: policies

To apply these changes you might need to restart kubelet.

sudo systemctl restart kubelet

OpenShift

OpenShift 4.x has audit log enabled by default. No additional configurations are required.

Our solution provides detailed Audit dashboards. By default, OpenShift does not provide audit logs. You can enable them by following instructions from OpenShift documentation Master and Node Configuration/Advanced Audit .

You need to enable audit log only on Masters. For that, you need to edit master-config.yaml file.

sudo vi /etc/origin/master/master-config.yaml

And add configuration, to keep for example audit for the last 10 days or maximum of 3 files with the size of 100 megabytes.

auditConfig:
    auditFilePath: "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log"
    enabled: true
    maximumFileRetentionDays: 10
    maximumFileSizeMegabytes: 100
    maximumRetainedFiles: 3
    policyFile: "/etc/origin/master/audit-policy.yaml"
    logFormat: json

Collector automatically forwards logs from /var/lib/origin/openpaas-oscp-audit/, no additional configuration is required in the collector.

We updated our configuration to use /var/lib/origin/openpaas-oscp-audit/ instead of /var/log recently. OpenShift starting from version 3.10 does not map /var/log inside the container with the master API.

You need to create an audit-policy.yaml in the specified path.

sudo vi /etc/origin/master/audit-policy.yaml

Use our example as a reference. We are hiding most of the system activity.

Another good example of the audit-policy.yaml file is an audit profile used by GCE.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # Do not log from kube-system accounts
  - level: None
    userGroups:
    - system:serviceaccounts:kube-system
    - system:serviceaccounts:openshift-infra
    - system:serviceaccounts:kube-service-catalog
    - system:serviceaccounts:openshift-template-service-broker
    - system:serviceaccounts:openshift-sdn
    - system:serviceaccounts:openshift-node
    - system:serviceaccounts:openshift-ansible-service-broker
    - system:serviceaccounts:openshift-monitoring
  - level: None
    users:
    - system:apiserver
    - system:kube-scheduler
    - system:volume-scheduler
    - system:kube-controller-manager
    - system:node
    - system:openshift-master

  # Do not log from collector
  - level: None
    users:
    - system:serviceaccount:collectorforopenshift:collectorforopenshift

  # Don't log nodes communications
  - level: None
    userGroups:
    - system:nodes

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
    - /healthz*
    - /version
    - /swagger*

  # Log configmap and secret changes in all namespaces at the metadata level.
  - level: Metadata
    resources:
    - resources: ["secrets", "configmaps"]

  # A catch-all rule to log all other requests at the request level.
  - level: Request

Restart services after making changes.

sudo /usr/local/bin/master-restart api
sudo /usr/local/bin/master-restart controllers

If you are using openshift 3.9 or below, the command to restart the services is sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers

Links

• Installation
  • Forwarding container logs, application logs, host logs and audit logs
  • Test our solution with the embedded 30 days evaluation license.
• Collector Configuration (Kubernetes)
  • Collector configuration reference for Kubernetes clusters.
• Collector Configuration (OpenShift)
  • Collector configuration reference for OpenShift clusters.
• Annotations
  • Changing type and format of messages forwarded from namespaces, workloads and pods.
  • Forwarding application logs.
  • Multi-line container logs.
  • Fields extraction for application and container logs (including timestamp extractions).
  • Hiding sensitive data, stripping terminal escape codes and colors.
• Audit Logs
  • Configure audit logs.
  • Forwarding audit logs.
• Troubleshooting
• FAQ and the common questions
• License agreement
• Pricing
• Contact