When more than one fleet of Docker hosts forwards data into the same Splunk, every dashboard turns into a guessing game — which environment is this container running in, which team owns it. The fix is to tag the data at the source so you can filter cleanly in Splunk.
Identifying the clusters
Identify the cluster with the configuration
The simplest tag is a cluster name. Set it once on each collectorfordocker container and every event from that fleet carries it:
1--env "COLLECTOR__CLUSTER=general__fields.docker_cluster=-"For example, on a development fleet:
1--env "COLLECTOR__CLUSTER=general__fields.docker_cluster=development"ACL for Clusters
When different teams should see different data — Team1 their containers, Team2 theirs, Admins everything — push the separation down to Splunk indexes and let Splunk’s role system enforce it. Every search in the app runs through macros, so once the indexes are wired up, the macros do the rest.
A worked example: you have Admins, Team1, and Team2. Admins should see Production and Development across the board, Team1 should see only the containers their team builds, and Team2 only theirs.
Define six indexes — one default per environment, plus one per team per environment:
docker_prod_team1docker_prod_team2docker_proddocker_dev_team1docker_dev_team2docker_dev
Then create two HEC tokens. The Production token defaults to docker_prod and is allowed to write to docker_prod_team1 and docker_prod_team2. The Development token defaults to docker_dev and is allowed to write to docker_dev_team1 and docker_dev_team2.
Use the Production token on Docker hosts in the production environment, the Development token on hosts in the development environment, and use annotations on the containers built by Team1 and Team2 to redirect their data into docker_prod_team1, docker_prod_team2, docker_dev_team1, and docker_dev_team2 respectively.
On the Splunk side, change the macros to search across index=docker_*, then create three roles: Admins with access to every index, Team1 with access to docker_prod_team1 and docker_dev_team1, and Team2 with access to docker_prod_team2 and docker_dev_team2. The same dashboards now show different data depending on who’s logged in — Team1 and Team2 see only logs and metrics from their own containers, while Admins see everything including system-level data.