Multiple clusters

When several clusters forward data into the same Splunk, tag each cluster’s data at the source so you can tell the clusters apart in the dashboards. To show each team only its own data, separate the data into indexes and let Splunk’s roles enforce it.

Without a tag, every dashboard turns into a guessing game: which cluster did this pod come from, and which environment is this node in?

Identify the cluster with the configuration

The simplest tag is a cluster name. Set it once on each Collectord deployment, and every event from that cluster carries it:

1[general]
2
3...
4
5fields.openshift_cluster = -

For example, on a development cluster:

1[general]
2
3...
4
5fields.openshift_cluster = development

Cluster labels

Node labels add a second axis, for filtering by environment, region or any other dimension your nodes already encode. Most dashboards in the app let you filter on them.

Take two clusters, prod and dev, each with the nodes master1, node1 and node2. Tag every node in the dev cluster with example.com/cluster: dev, and every node in prod with example.com/cluster: prod. Edit each node with oc:

1$ oc edit nodes/master1

Find the labels list and append the new label:

1  labels:
2    beta.kubernetes.io/arch: amd64
3    beta.kubernetes.io/os: linux
4    kubernetes.io/hostname: master1
5    node-role.kubernetes.io/master: ""
6    example.com/cluster: dev

Repeat for every node in every cluster. The dashboards then let you filter on example.com/cluster=dev or example.com/cluster=prod.

Collectord reads node labels at startup only. After changing labels, restart Collectord so it picks them up.

ACL for Clusters

When different teams should see different data, push the separation down to Splunk indexes and let Splunk’s role system enforce it. Every search in the app runs through macros, so once the indexes are wired up, the macros do the rest.

For example, take three groups of users, each with its own view of the clusters:

• Admins see Production and Development across all namespaces.
• Team1 sees only NamespaceTeam1.
• Team2 sees only NamespaceTeam2.

To set it up:

1. Define six indexes, one default per cluster plus one per team per cluster:
   • openshift_prod_team1
   • openshift_prod_team2
   • openshift_prod
   • openshift_dev_team1
   • openshift_dev_team2
   • openshift_dev
2. Create two HEC tokens:
   • The Production token defaults to openshift_prod and is allowed to write to openshift_prod_team1 and openshift_prod_team2.
   • The Development token defaults to openshift_dev and is allowed to write to openshift_dev_team1 and openshift_dev_team2.
3. Use the Production token on the production cluster, and the Development token on the development cluster.
4. Use annotations on NamespaceTeam1 and NamespaceTeam2 to redirect each team’s data into its index on that cluster: openshift_prod_team1 and openshift_prod_team2 on production, openshift_dev_team1 and openshift_dev_team2 on development.
5. On the Splunk side, change the macros to search across index=openshift_*.
6. Create three roles:
   • Admins, with access to every index;
   • Team1, with access to openshift_prod_team1 and openshift_dev_team1;
   • Team2, with access to openshift_prod_team2 and openshift_dev_team2.

The same dashboards now show different data depending on who’s logged in. Team1 and Team2 see only logs and metrics from their own pods, while Admins see everything, system-level data included.