Outcold Solutions LLC

Monitoring OpenShift - Version 4

You are looking at documentation for an older release. See the current release documentation.

Configuration

To find server version of your OpenShift cluster use

$ oc version

Match the server version with one of the links below. If you are using version, which is not in this list you can try the closest version.

If you are using version, which is not on the list, please email us contact@outcoldsolutions.com

Using images on hub.docker.com

These images are build on top of Alpine linux

Using certified images on registry.connect.redhat.com

These images are built on top of RHEL images, see outcoldsolutions/collectorforopenshift. To pull images from this registry you need to authenticate, see instructions below.

registry.connect.redhat.com authentication

registry.connect.redhat.com is not the same as registry.access.redhat.com, second is used for Red Hat images, first is used for certified images from partners. Second works with OpenShift cluster out of the box, first requires authentication.

You need to specify secret to authenticate with registry.connect.redhat.com. Please follow the link to learn how to use other secured registries. Allowing Pods to Reference Images from Other Secured Registries

This is an example how you can authenticate with registry.connect.redhat.com.

After applying the configuration to your OpenShift cluster you need to create a secret for pulling images from Red Hat registry. Make sure you are in the same project/namespace as where collector is created (collectorforopenshift is a default project/namespace).

$ oc project collectorforopenshift

If you are on Linux (for macOS see below), you can login to the registry using docker and use authentication file to create a secret in OpenShift cluster.

$ docker login registry.connect.redhat.com
Username: [redhat-username]
Password: [redhat-user-password]
Login Succeeded

Make sure to use username and not email, when you login to this registry. They both allows you to login. But if you logged in with email, you will not be able to download the image.

After that you can create a secret for pulling images using just created authentication under $HOME/.docker/config.json

$ oc --namespace collectorforopenshift secrets new rhcc .dockerconfigjson=$HOME/.docker/config.json

On macOS Docker does not store authentication data in config.json (stores in keychain instead). You cannot use it to create a secret. Instead you can create a secret from the command line with oc secrets new-dockercfg rhcc --docker-server=registry.connect.redhat.com --docker-username=<user_name> --docker-password=<password> --docker-email=<email>. Just make sure this command is not going to be saved in the bash history, as it is going to have a password in the command line. See Execute command without keeping it in history. You can execute export HISTFILE=/dev/null in this terminal session, which will stop recording any commands in history.

Link created secret rhcc to the service account we use for collector collectorforopenshift

$ oc --namespace collectorforopenshift secrets link collectorforopenshift rhcc --for=pull

If some pods have been created before you linked the secret you will need to recreate them. You can delete all the pods under collectorforopenshift namespace, and scheduler will recreate pods with right secret for pulling images.

oc delete --namespace collectorforopenshift pods --all

Created OpenShift Objects

Configuration file collectorforkubernetes.yaml creates several OpenShift Objects.

  • Project collectorforopenshift.
  • ClusterRole collectorforopenshift with limited capabilities to get, list and watch most of the various deployment objects. Collector uses this information to enrich logs and stats with openshift specific metadata.
  • ServiceAccount collectorforopenshift is used to connect to OpenShift API.
  • ClusterRoleBinding collectorforopenshift to bind service account to cluster role.
  • ConfigMap collectorforopenshift delivers configuration file for collector.
  • DaemonSet collectorforopenshift allows to deploy collector on none-master nodes.
  • DaemonSet collectorforopenshift-master allows to deploy collector on master nodes.
  • Deployment collectorforopenshift-addon is a single collector, that needs to forward data from the whole cluster once.

Please read commentaries in collectorforopenshift.yaml file to get more deep details on all configurations and source of the logs and metrics.

Collector configuration

ConfigMap collectorforopenshift delivers configuration file for collector. This is an ini file, where you can find all the default values.

Values can be overridden using environment values with the format as specified below

COLLECTOR__{ANY_NAME}={section}__{key}={value}

Configurations with environment variables are the simplest way to explore and debug quickly, but we recommend to write your configuration file based on the default provided with collectorforopenshift.yaml.

Join Rules

By default collector joins all messages with previous if they start with spaces. Below you can find how to specify a custom rule on the example of java application.

If this is a sample of the application logs.

[2017-09-04T06:28:05,664][WARN ][MyComponent]
java.security.AccessControlException: access denied
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
  at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
[2017-09-04T06:28:05,664][WARN ][MyComponent] another message

You can specify the join rules, where you configure that you want to match all containers with the name that contains my_app in their name, and pattern for the new message should match regex ^\[\d{4}-.

[pipe.join::my_app]
matchRegex.openshift_container_name = .+my_app.+
patternRegex = ^\[\d{4}-

Cluster labels

Our dashboards allows you to filter nodes based on the node labels, if you label your nodes with the names of the clusters you can easily filter in into specific cluster. To leverage that capability, you need to label nodes in your clusters.

For example, if you have two clusters prod and dev, each cluster has master1, node1 and node2 nodes you can apply labels to every node by using kubectl.

As an example for dev cluster, node master you can append label example.com/cluster: dev

$ oc edit nodes/master1

And add label

1
2
3
4
5
6
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/hostname: master1
    node-role.kubernetes.io/master: ""
    example.com/cluster: dev

Do that for all of the nodes in your clusters. After that you will be able to filter clusters in dashboards by using node labels example.com/cluster=dev and example.com/cluster=prod.

Our collector reads node labels only at start. To apply this change to collector you need to restart it. One of doing that is by applying the label to the pod template in daemonset configuration.


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.