Access control

Two independent questions govern who sees what:

1. Can this Splunk user run the commands at all? Answered by Splunk roles and capabilities.
2. Whose Kubernetes permissions does the search run under? Answered by the credential model you choose for each cluster.

Get both right and Kubernetes Search gives analysts controlled, audited read access to clusters without handing out kubectl or cluster credentials.

Splunk roles and capabilities

The app defines two capabilities:

And two roles that bundle them:

These are the names Splunk shows under Settings - Roles; in authorize.conf the stanzas are [role_k8s_search_user] and [role_k8s_search_admin]. A full Splunk administrator (admin_all_objects) can do everything without an explicit role. Neither role gets list_storage_passwords: search commands resolve their credential through a system-context handler, so users run searches without being able to read the credential store directly, and the administrator role manages credentials through the app’s own handlers. That role manages every cluster’s credentials, so grant it on the same basis you would grant the ability to issue those clusters’ service-account tokens. What it cannot do is read back a credential the design keeps from it: an impersonation cluster’s service account, or the credential a minting cluster requests its tokens with. Neither of the two sequences that used to allow that works any more. Changing a cluster’s impersonation setting while its credentials are stored is refused (see below), and re-pointing a cluster’s server is allowed but leaves its stored credentials behind: they stay bound to the server they were stored for and stop resolving until they are entered again (see Clusters).

edit_k8s_search_clusters also works on its own, in a role of yours without run_k8s_search: for someone who keeps the cluster registry and its credentials without being allowed to query the clusters in it. The Setup page loads and saves for such a role, which needs nothing else from Splunk - not even the stock user role. Splunk reads these permissions when it starts, so after an upgrade from 1.0.0 this works once splunkd has restarted.

Credential models

Each cluster uses one of four models. They differ in whose Kubernetes RBAC a search runs under and in how well one user is isolated from another. This matters most on clusters where different people should see different things.

Shared default credential

One credential is configured for the cluster and every Splunk user’s searches use it. Simplest to set up; no per-user configuration. The cluster form says so next to the credential fields.

The shared credential is extractable

Any Splunk user who can run searches against the cluster can arrange to read the token or key and use it outside Splunk with its full Kubernetes RBAC, from anywhere that can reach the API server, until you rotate it. The app returns it only for a running search of the user’s own, which stops a one-line | rest call but not a user who starts a search and presents it. Treat a shared credential as readable by everyone who has run_k8s_search, and scope it to least privilege accordingly - read-only, no Secrets. It is the right choice for a low-sensitivity, read-only cluster where everyone is entitled to the same view; for any other cluster, use one of the models below.

Minted tokens

The cluster’s stored credential never reaches a search. It only requests a ten-minute token for a named, read-only service account through the Kubernetes TokenRequest API, and every search authenticates with such a token. Users still share the service account’s view, as with a shared credential, but no user can read the stored credential back through the app, and a token a user takes out of Splunk expires within minutes and cannot be renewed once their Splunk access is gone. Each token is recorded with the Splunk user it was issued to. Minting is available on every license tier. The RBAC it needs and how to set it up are in Clusters - Minted tokens.

Per-user credentials

Each Splunk user registers their own credential for the cluster - a bearer token or a client certificate - under their own account. A user’s credential is isolated from other users - the app refuses cross-user reads - so people query with their own Kubernetes RBAC. A user can still extract their own credential (it is theirs), but not anyone else’s. Use this when users genuinely have different Kubernetes permissions and you want each search to reflect the searcher’s own access. Per-user credentials require a paid license.

Impersonation

The cluster holds one service-account credential that is allowed to impersonate users. At search time the app stamps the Splunk username into the request’s Impersonate-User header, and the Kubernetes API server applies that user’s RBAC. The service-account bearer never reaches the search process, and each user is limited to exactly what their Kubernetes identity allows. This is the strongest isolation. Impersonation requires a paid license.

Set it up carefully:

• The service account needs the impersonate verb on users only - never on groups. Kubernetes Search stamps the user, not groups.
• Bind your Kubernetes RBAC to Splunk usernames, since that is what gets impersonated.
• A search with no real Splunk-user identity - a scheduled or saved search runs as nobody - is refused impersonation. Impersonation is for interactive analyst queries.
• Turning impersonation on or off is refused while the cluster still stores its default credentials, in both directions: delete them first (DELETE /k8s_search/clusters/<name>/auth), change the setting, then store the credential the new mode needs. Switching it on with a shared credential in place would hand everyone who could read that credential the ability to act as any user; switching it off with the service account in place would expose the account that may impersonate them.
• Server-side impersonation queries running at once on a search head are capped. Over the cap a query is refused with a message asking the client to retry, rather than the search head forking without limit.

Choosing a model

As a rule: minted tokens for a cluster everyone may see equally but whose credential should stay on the search head, shared only for a low-sensitivity cluster where you cannot grant the RBAC minting needs, impersonation for sensitive or multi-tenant clusters where the API server should be the one enforcing per-user access, and per-user when impersonation isn’t available but users still need distinct RBAC.

Auditing usage

Because Kubernetes Search gives live read access to clusters, you’ll want to see who ran what, and who changed the configuration behind it. There are three audit trails.

In Splunk. Every search is recorded in the audit log, with the user and the SPL they ran, so you can review Kubernetes Search activity directly:

index=_audit action=search
| search search="*| k8s*" OR search="*k8slogs*" OR search="*k8sevents*" OR search="*k8sdescribe*"
| table _time user search

In the app’s own trail. Splunk’s audit index does not say which cluster a search queried or which credential it resolved, and some of the app’s calls have no search job behind them. So Kubernetes Search writes one k8s_search audit: line to _internal for every change made on the Setup page, every server-side query and credential resolution, and every minted token. A line never carries a credential, token or license key value - a field the request wrote appears by name only.

index=_internal "k8s_search audit:"

An exec is recorded before it runs, so one that hangs or is killed at the timeout is still on record - with the kind and namespace it asked for, not the pod, container or event filters. A credential resolution is recorded once the credential has been handed out; a call refused by the impersonation or server-binding gates leaves no line here, only its refusal. A config_change with result=refused always means nothing was changed. To review who changed the configuration:

index=_internal "k8s_search audit:" event=config_change
| table _time user method object keys result

In Kubernetes. With impersonation, each API request reaches the cluster under the impersonated Splunk username, so the cluster’s own API-server audit log attributes activity to the individual user - per-user accountability in the cluster’s trail, not just in Splunk. With a shared credential, requests appear under the single shared identity instead, which is one more reason to prefer impersonation where attribution matters. With minted tokens, requests appear under the search service account, but on Kubernetes 1.30 and later (1.29 with the ServiceAccountTokenJTI feature gate on) each one carries the token’s credential id (authentication.kubernetes.io/credential-id), which the app’s event=token line ties to the Splunk user it was issued to.