Monitoring OpenShift

Audit logs

Learn how to configure and forward OpenShift audit logs for monitoring

OpenShift 4.x enables audit logging by default — no extra configuration needed. The rest of this page applies to OpenShift 3.x.

The Monitoring OpenShift app ships dashboards built specifically for API server audit data — who did what, against which resource, and how the API responded. On OpenShift 3.x, audit logs aren’t on by default, so you’ll need to enable them at the master before Collectord can forward them. For background, see the OpenShift Master and Node Configuration / Advanced Audit guide.

Enable auditing on each master by editing master-config.yaml:

bash
1sudo vi /etc/origin/master/master-config.yaml

The block below keeps audit data for 10 days, capped at 3 rotated files of 100 MB each — adjust to fit your retention policy and disk budget:

master-config.yaml yaml
1auditConfig:
2  auditFilePath: "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log"
3  enabled: true
4  maximumFileRetentionDays: 10
5  maximumFileSizeMegabytes: 100
6  maximumRetainedFiles: 3
7  policyFile: "/etc/origin/master/audit-policy.yaml"
8  logFormat: json

Collectord forwards anything written under /var/lib/origin/openpaas-oscp-audit/ automatically — no Collectord-side changes are required.

Older versions of this guide pointed at /var/log. Starting with OpenShift 3.10, /var/log is no longer mapped into the master API container, which is why the path moved to /var/lib/origin/openpaas-oscp-audit/.

Next, create the policy file:

bash
1sudo vi /etc/origin/master/audit-policy.yaml

The example below silences the noisiest system traffic — kube-system, openshift-infra, the SDN, the monitoring stack, Collectord itself — logs secret and configmap mutations at the metadata level, and catches everything else at the request level. Use it as a starting point.

For a more exhaustive policy, see the audit profile used by GCE.

audit-policy.yaml yaml
 1apiVersion: audit.k8s.io/v1beta1
 2kind: Policy
 3rules:
 4  # Do not log from kube-system accounts
 5  - level: None
 6    userGroups:
 7    - system:serviceaccounts:kube-system
 8    - system:serviceaccounts:openshift-infra
 9    - system:serviceaccounts:kube-service-catalog
10    - system:serviceaccounts:openshift-template-service-broker
11    - system:serviceaccounts:openshift-sdn
12    - system:serviceaccounts:openshift-node
13    - system:serviceaccounts:openshift-ansible-service-broker
14    - system:serviceaccounts:openshift-monitoring
15  - level: None
16    users:
17    - system:apiserver
18    - system:kube-scheduler
19    - system:volume-scheduler
20    - system:kube-controller-manager
21    - system:node
22    - system:openshift-master
23
24  # Do not log from collector
25  - level: None
26    users:
27    - system:serviceaccount:collectorforopenshift:collectorforopenshift
28
29  # Don't log nodes communications
30  - level: None
31    userGroups:
32    - system:nodes
33
34  # Don't log these read-only URLs.
35  - level: None
36    nonResourceURLs:
37    - /healthz*
38    - /version
39    - /swagger*
40
41  # Log configmap and secret changes in all namespaces at the metadata level.
42  - level: Metadata
43    resources:
44    - resources: ["secrets", "configmaps"]
45
46  # A catch-all rule to log all other requests at the request level.
47  - level: Request

Restart the master services so the new configuration takes effect:

bash
1sudo /usr/local/bin/master-restart api
2sudo /usr/local/bin/master-restart controllers

On OpenShift 3.9 or older, restart with sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers instead.

The Splunk app finds audit events through the macro_openshift_audit_logs macro, which scopes searches to host logs containing the audit.k8s.io API group:

text
1(`macro_openshift_host_logs` "audit.k8s.io")