Multiple clusters

When several fleets of Docker hosts forward data into the same Splunk, tag each fleet’s data at the source so you can tell the fleets apart in the dashboards. To show each team only its own data, separate the data into indexes and let Splunk’s roles enforce it.

Without a tag, every dashboard turns into a guessing game: which environment is this container running in, and which team owns it?

Identifying the clusters

Identify the cluster with the configuration

The simplest tag is a cluster name. Set it once on each collectorfordocker container, and every event from that fleet carries it:

1--env "COLLECTOR__CLUSTER=general__fields.docker_cluster=-"

For example, on a development fleet:

1--env "COLLECTOR__CLUSTER=general__fields.docker_cluster=development"

ACL for Clusters

When different teams should see different data, push the separation down to Splunk indexes and let Splunk’s role system enforce it. Every search in the app runs through macros, so once the indexes are wired up, the macros do the rest.

For example, take three groups of users, each with its own view of the fleets:

• Admins see Production and Development across the board.
• Team1 sees only the containers its team builds.
• Team2 sees only its own.

To set it up:

1. Define six indexes, one default per environment plus one per team per environment:
   • docker_prod_team1
   • docker_prod_team2
   • docker_prod
   • docker_dev_team1
   • docker_dev_team2
   • docker_dev
2. Create two HEC tokens:
   • The Production token defaults to docker_prod and is allowed to write to docker_prod_team1 and docker_prod_team2.
   • The Development token defaults to docker_dev and is allowed to write to docker_dev_team1 and docker_dev_team2.
3. Use the Production token on Docker hosts in the production environment, and the Development token on hosts in the development environment.
4. Label the containers Team1 and Team2 build with container annotations that redirect each team’s data into its index for that environment: docker_prod_team1 and docker_prod_team2 in production, docker_dev_team1 and docker_dev_team2 in development.
5. On the Splunk side, change the macros to search across index=docker_*.
6. Create three roles:
   • Admins, with access to every index;
   • Team1, with access to docker_prod_team1 and docker_dev_team1;
   • Team2, with access to docker_prod_team2 and docker_dev_team2.

The same dashboards now show different data depending on who’s logged in. Team1 and Team2 see only logs and metrics from their own containers, while Admins see everything, system-level data included.