Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Kubernetes

Monitoring Multiple Clusters

Identify the cluster with the configuration

When you apply the collectorforkubernetes configuration, specify the cluster name with the configuration

[general]

...

fields.kubernetes_cluster = -

For example

[general]

...

fields.kubernetes_cluster = development

Cluster labels

Our dashboards allow you to filter nodes based on the node labels.

If you have two clusters prod and dev, and each cluster has master1, node1 and node2 nodes, you can apply labels to every node with kubectl.

As an example, in the dev cluster for the node master1 you can append the label example.com/cluster: dev.

$ kubectl edit nodes/master1

Find labels list and append new label.

  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/hostname: master1
    node-role.kubernetes.io/master: ""
    example.com/cluster: dev

If you do that for all of the nodes in all of your clusters, you will be able to use these labels on most of the dashboards of our applications. With the given example, you will be able to filter by labels example.com/cluster=dev and example.com/cluster=prod.

Our collectord reads node labels only at the start. To apply this change to the collectord you need to restart it.

ACL for Clusters

All searches in the application are powered by macros. If you want to separate access to the data for specific clusters or namespaces, you can define different target indexes for clusters or namespaces and update the macros to use these indexes.

For example, let’s assume you have Admins, Team1, and Team2 organizations in your company. You want Admins to see data from Production and Development environments and all namespaces, Team1 to see only data from NamespaceTeam1, and Team2 to see only data from NamespaceTeam2.

You can define several indices

  • kubernetes_prod_team1
  • kubernetes_prod_team2
  • kubernetes_prod
  • kubernetes_dev_team1
  • kubernetes_dev_team2
  • kubernetes_dev

Create two HTTP Tokens. One for the Production cluster with the default index kubernetes_prod—allow this Token to write to kubernetes_prod_team1 and kubernetes_prod_team2. Another token for the Development cluster with the default index kubernetes_dev—allow this Token to write to kubernetes_dev_team1 and kubernetes_dev_team2.

For the Kubernetes cluster running in the Production environment, use the first token; for the cluster running in the Development environment, use the second token. Use annotations to override indexes for namespaces NamespaceTeam1 and NamespaceTeam2 to redirect their data to indexes kubernetes_prod_team1, kubernetes_prod_team2, kubernetes_dev_team1, and kubernetes_dev_team2.

In Splunk, change the macros to always search in the indices index=kubernetes_*. Create 3 roles in Splunk: one for Admins that has access to all created indices, a second role for Team1 with access to kubernetes_prod_team1 and kubernetes_dev_team1, and a third role for Team2 with access to kubernetes_prod_team2 and kubernetes_dev_team2. Now, depending on who is logged in to Splunk, you will see a different set of data in the application. Team1 and Team2 will not be able to see system-related information— only logs and metrics from their Pods running in their namespaces. Admins will be able to see all the information.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.