Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Pricing Contact Us Blog Evaluation licenses Partners License Agreement Privacy Policy FAQ

Monitoring OpenShift

Forwarding Audit Logs

OpenShift 4.x has audit log enabled by default. No additional configurations are required.

Our solution provides detailed Audit dashboards. By default, OpenShift does not provide audit logs. You can enable them by following the instructions from the OpenShift documentation Master and Node Configuration/Advanced Audit.

You need to enable audit logs only on Masters. For that, you need to edit the master-config.yaml file.

1sudo vi /etc/origin/master/master-config.yaml

And add configuration to keep, for example, audit logs for the last 10 days or a maximum of 3 files with the size of 100 megabytes.

1auditConfig:
2  auditFilePath: "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log"
3  enabled: true
4  maximumFileRetentionDays: 10
5  maximumFileSizeMegabytes: 100
6  maximumRetainedFiles: 3
7  policyFile: "/etc/origin/master/audit-policy.yaml"
8  logFormat: json

The collectord automatically forwards logs from /var/lib/origin/openpaas-oscp-audit/; no additional configuration is required in the collectord.

We recently updated our configuration to use /var/lib/origin/openpaas-oscp-audit/ instead of /var/log. OpenShift, starting from version 3.10, does not map /var/log inside the container with the master API.

You need to create an audit-policy.yaml in the specified path.

1sudo vi /etc/origin/master/audit-policy.yaml

Use our example as a reference. We hide most of the system activity.

Another good example of the audit-policy.yaml file is the audit profile used by GCE.

 1apiVersion: audit.k8s.io/v1beta1
 2kind: Policy
 3rules:
 4  # Do not log from kube-system accounts
 5  - level: None
 6    userGroups:
 7    - system:serviceaccounts:kube-system
 8    - system:serviceaccounts:openshift-infra
 9    - system:serviceaccounts:kube-service-catalog
10    - system:serviceaccounts:openshift-template-service-broker
11    - system:serviceaccounts:openshift-sdn
12    - system:serviceaccounts:openshift-node
13    - system:serviceaccounts:openshift-ansible-service-broker
14    - system:serviceaccounts:openshift-monitoring
15  - level: None
16    users:
17    - system:apiserver
18    - system:kube-scheduler
19    - system:volume-scheduler
20    - system:kube-controller-manager
21    - system:node
22    - system:openshift-master
23
24  # Do not log from collector
25  - level: None
26    users:
27    - system:serviceaccount:collectorforopenshift:collectorforopenshift
28
29  # Don't log nodes communications
30  - level: None
31    userGroups:
32    - system:nodes
33
34  # Don't log these read-only URLs.
35  - level: None
36    nonResourceURLs:
37    - /healthz*
38    - /version
39    - /swagger*
40
41  # Log configmap and secret changes in all namespaces at the metadata level.
42  - level: Metadata
43    resources:
44    - resources: ["secrets", "configmaps"]
45
46  # A catch-all rule to log all other requests at the request level.
47  - level: Request

Restart the services after making changes.

1sudo /usr/local/bin/master-restart api
2sudo /usr/local/bin/master-restart controllers

If you are using OpenShift 3.9 or below, the command to restart the services is sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers

The application has a macro that defines how to find the audit logs: macro_openshift_audit_logs.

1(`macro_openshift_host_logs` "audit.k8s.io")

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.

Red Hat
Splunk
AWS