You are looking at documentation for an older release. See the current release documentation.
Installation
With our solution for Monitoring OpenShift, you can start monitoring your clusters in under 10 minutes, including forwarding metadata-enriched container logs, host logs, and metrics.
Features:
- Log forwarding is built on native JSON logging driver.
- Tiny image, tiny binary. Very low memory, CPU, and disk consumption.
- Logs are enriched with OpenShift metadata (container, image, pod, daemon sets, jobs, cron jobs, etc.).
- Collect stats and events, allowing you to correlate logs with metrics.
- Collects process metrics.
- Forwards host logs allows us to monitor main components of the cluster.
- Uses HTTP Event Collector to ingest data in Splunk. Requires Splunk version 6.5 or above (talk to us if you need support for an earlier version of Splunk).
- Multi-line events support.
- Security Monitoring (privileged containers and enabled capabilities).
- Capacity Monitoring (Available and allocated resource, CPU Shares, CPU Quotas, Memory Limits for containers).
- At least once delivery guarantee.
Splunk configuration
Install Monitoring OpenShift application
Install Monitoring OpenShift from splunkbase. You need to install it on Search Heads only.
Enable HTTP Event Collector in Splunk
Outcold Solutions' Collector sends data to Splunk using HTTP Event Collector. By default, Splunk does not enable HTTP Event Collector. Please read HTTP Event Collector walkthrough to learn more about HTTP Event Collector.
After enabling HTTP Event Collector, you need to find correct Url for HTTP Event Collector and generate an HTTP Event Collector Token.
If you are running your Splunk instance on hostname hec.example.com
,
it listens on port 8088
, using SSL
and token is B5A79AAD-D822-46CC-80D1-819F80D7BFB0
you can test it with the curl
command as in the example below.
$ curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}' {"text": "Success", "code": 0}
-k
is necessary for self-signed certificates.
OpenShift preparation
To be able to use our solution and get all the benefits, you will need to perform preparation on every OpenShift node in your cluster.
Docker logging driver
When you set up your OpenShift cluster, verify that docker uses json-file
logging driver.
RHEL by default configures docker with journald
. Base on your
Linux distribution you can find this configuration in various places. In case of
latest RHEL Server 7.4 you can find it under /etc/sysconfig/docker
.
Replace --log-driver=journald
with
--log-driver=json-file --log-opt max-size=1M --log-opt max-file=3
.
It is important to limit the size of the log files and number of them, see
Managing Container Logs
for details.
$ sed -i 's/--log-driver=journald/--log-driver=json-file --log-opt max-size=1M --log-opt max-file=3/' /etc/sysconfig/docker
$ systemctl restart docker
If you are using
Red Hat Container Development Kit, it will pre-setup minishift with journald
logging driver. You can change it when you start minishift for the first time
with minishift start --docker-opt log-driver=json-file
.
Syslog and host logs
RHEL Server distribution might not include rsyslog installed by default.
That means that all host logs (including OpenShift components and host logs) are
stored only with journalctl
. By default, journalctl
knows how to forward logs to
local syslog server. In most cases you just need to install rsyslog
and after that, you will see host logs
under /var/log/messages
.
$ sudo yum install rsyslog
$ sudo systemctl enable rsyslog
$ sudo systemctl start rsyslog
Verify that you can see syslog messages under /var/log/messages
.
$ tail /var/log/messages
Install Collector for OpenShift
Verify that you are in the context of the user who can perform admin operations (cluster-admin
role).
$ oc login -u system:admin
Because we are monitoring log files located on the host volumes, collector needs access to host volume. To enable it run (see Use the hostPath Volume Plug-in)
$ oc edit scc privileged
Verify that privileged
security context can access host dir volumes and can run privileged containers.
By default it has these capabilities.
allowHostDirVolumePlugin: true allowPrivilegedContainer: true
Use latest OpenShift configuration file collectorforopenshift.yaml. This configuration deploys latest image of collectorforopenshift on all OpenShift nodes as a DaemonSet workload.
Open it in your favorite editor and set the Splunk HTTP Event Collector Url, token, configuration for a certificate if required, and accept a license agreement and include license key (request an evaluation license key with this automated form).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | [general] ; acceptEULA = false ; license = ... # Splunk output [output.splunk] # Splunk HTTP Event Collector url ; url = # Splunk HTTP Event Collector Token ; token = # Allow invalid SSL server certificate ; insecure = false # Path to CA certificate ; caPath = # CA Name to verify ; caName = |
Based on the example above you will need to modify the lines as in the following.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | [general] acceptEULA = true license = ... ... # Splunk output [output.splunk] # Splunk HTTP Event Collector url url = https://hec.example.com:8088/services/collector/event/1.0 # Splunk HTTP Event Collector Token token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0 # Allow invalid SSL server certificate insecure = true |
Apply this change to your OpenShift cluster with oc
$ oc apply -f ./collectorforopenshift.yaml
After that, you need to add privileged security context to the Service Account we use for the collector.
$ oc adm policy add-scc-to-user privileged system:serviceaccount:default:collectorforopenshift
If you see an error message
the server could not find the requested resource
, possible that you are using a mismatched version of theoc
tool and the server version. You can accomplish the same by using commandoc edit securitycontextconstraints privileged
and addingsystem:serviceaccount:default:collectorforopenshift
to the list ofusers
.
If you are using Red Hat certified images from registry.connect.redhat.com
, make sure to specify the secret for pulling
the image. See instructions on the Configuration Reference page.
Verify that Daemon Set is deployed.
$ oc get daemonsets --all-namespaces
If collectorforopenshift
Pods aren't deployed, follow the
Troubleshooting steps.
Give it a few moments to download the image and start the container. After all the pods are deployed, go to the Monitoring OpenShift application in Splunk and you should see data on dashboards.