Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Forwarding Kubernetes logs to QRadar (syslog)

Configuration"

Using images on docker.io (hub.docker.com)

These images are built on top of Scratch images.

collectorforopenshift-syslog.yaml

Using certified images on registry.connect.redhat.com

These images are built on top of RHEL images, see outcoldsolutions/collectorforopenshift. To pull images from this registry you need to authenticate, see instructions below.

collectorforopenshift-rhel-syslog.yaml

registry.connect.redhat.com authentication

registry.connect.redhat.com is not the same as registry.access.redhat.com, second is used for Red Hat images, first is used for certified images from partners. Second works with OpenShift cluster out of the box, first requires authentication.

You need to specify secret to authenticate with registry.connect.redhat.com. Please follow the link to learn how to use other secured registries. Allowing Pods to Reference Images from Other Secured Registries

This is an example how you can authenticate with registry.connect.redhat.com.

After applying the configuration to your OpenShift cluster you need to create a secret for pulling images from Red Hat registry. Make sure you are in the same project/namespace as where collectord is created (collectorforopenshift-syslog is a default project/namespace).

$ oc project collectorforopenshift-syslog

If you are on Linux (for macOS see below), you can login to the registry using docker and use authentication file to create a secret in OpenShift cluster.

$ docker login registry.connect.redhat.com
Username: [redhat-username]
Password: [redhat-user-password]
Login Succeeded

Make sure to use username and not email, when you login to this registry. They both allows you to login. But if you logged in with email, you will not be able to download the image.

After that you can create a secret for pulling images using just created authentication under $HOME/.docker/config.json

$ oc --namespace collectorforopenshift-syslog secrets new rhcc .dockerconfigjson=$HOME/.docker/config.json

On macOS Docker does not store authentication data in config.json (stores in keychain instead). You cannot use it to create a secret. Instead you can create a secret from the command line with oc secrets --namespace collectorforopenshift-syslog new-dockercfg rhcc --docker-server=registry.connect.redhat.com --docker-username=<user_name> --docker-password=<password> --docker-email=<email>. Just make sure this command is not going to be saved in the bash history, as it is going to have a password in the command line. See Execute command without keeping it in history. You can execute export HISTFILE=/dev/null in this terminal session, which will stop recording any commands in history.

Link created secret rhcc to the service account we use for collectord collectorforopenshift-syslog

$ oc --namespace collectorforopenshift-syslog secrets link collectorforopenshift-syslog rhcc --for=pull

If some pods have been created before you linked the secret you will need to recreate them. You can delete all the pods under collectorforopenshift-syslog namespace, and scheduler will recreate pods with right secret for pulling images.

oc delete --namespace collectorforopenshift-syslog pods --all

Created OpenShift Objects

Configuration file collectorforopenshift-syslog.yaml creates several OpenShift Objects.

  • Project collectorforopenshift-syslog.
  • ClusterRole collectorforopenshift-syslog with limited capabilities to get, list and watch most of the various deployment objects. Collectord uses this information to enrich logs and stats with openshift specific metadata.
  • ServiceAccount collectorforopenshift-syslog is used to connect to OpenShift API.
  • ClusterRoleBinding collectorforopenshift-syslog to bind service account to cluster role.
  • ConfigMap collectorforopenshift-syslog delivers configuration file for collectord.
  • DaemonSet collectorforopenshift-syslog allows to deploy collectord on none-master nodes.
  • DaemonSet collectorforopenshift-syslog-master allows to deploy collectord on master nodes.
  • Deployment collectorforopenshift-syslog-addon is a single collectord, that needs to forward data from the whole cluster once.

Please read commentaries in collectorforopenshift-syslog.yaml file to get more deep details on all configurations and source of the logs and metrics.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.