Outcold Solutions LLC

Monitoring Kubernetes - Version 4

Configuration

To find server version of your Kubernetes cluster use

$ kubectl version

Match the server version with one of the links below. If you are using version, which is not in this list you can try the closest version.

If you are using version, which is not on the list, please email us contact@outcoldsolutions.com

Created Kubernetes Objects

Configuration file collectorforkubernetes.yaml creates several Kubernetes Objects.

  • Namespace collectorforopenshift.
  • ClusterRole collectorforkubernetes with limited capabilities to get, list and watch deployed objects. Collector uses this information to enrich logs and stats with kubernetes specific metadata.
  • ServiceAccount collectorforkubernetes is used to connect to Kubernetes API.
  • ClusterRoleBinding collectorforkubernetes to bind service account to cluster role.
  • ConfigMap collectorforkubernetes delivers configuration file for collector.
  • DaemonSet collectorforkubernetes allows to deploy collector on none-master nodes.
  • DaemonSet collectorforkubernetes-master allows to deploy collector on master nodes.
  • Deployment collectorforkubernetes-addon is a single collector, that needs to forward data from the whole cluster once.

Read commentaries in collectorforkubernetes.yaml file to get more deep details on all configurations and source of the logs and metrics.

Collector configuration

ConfigMap collectorforkubernetes delivers configuration file for collector. This is an ini file, where you can find all the default values.

Values can be overridden using environment values with the format as specified below

COLLECTOR__{ANY_NAME}={section}__{key}={value}

Configurations with environment variables are the simplest way to explore and debug quickly, but we recommend to write your configuration file based on the default provided with collectorforkubernetes.yaml.

Join Rules

By default collector joins all messages with previous if they start with spaces. Below you can find how to specify a custom rule on the example of java application.

If this is a sample of the application logs.

[2017-09-04T06:28:05,664][WARN ][MyComponent]
java.security.AccessControlException: access denied
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
  at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
[2017-09-04T06:28:05,664][WARN ][MyComponent] another message

You can specify the join rules, where you configure that you want to match all containers with the name that contains my_app in their name, and pattern for the new message should match regex ^\[\d{4}-.

[pipe.join::my_app]
matchRegex.kubernetes_container_name = .+my_app.+
patternRegex = ^\[\d{4}-

Cluster labels

Our dashboards allows you to filter nodes based on the node labels.

If you have two clusters prod and dev, each cluster has master1, node1 and node2 nodes you can apply labels to every node with kubectl.

As an example, in the dev cluster for the node master you can append label example.com/cluster: dev.

$ kubectl edit nodes/master1

Find labels list and append new label.

1
2
3
4
5
6
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/hostname: master1
    node-role.kubernetes.io/master: ""
    example.com/cluster: dev

If you do that for all of the nodes in all of your clusters, you will be able to use these labels on most of the dashboards of our applications. With the given example, you will be able to filter by labels example.com/cluster=dev and example.com/cluster=prod.

Our collector reads node labels only at the start. To apply this change to the collector you need to restart it.


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which gives you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which helps developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.