Clusters

Before you can query a cluster you register it once on the Setup page. A registration is three things: where the API server is, how to trust its certificate, and a credential to authenticate with. This page covers how to add clusters and the authentication options.

Add a cluster

Open the app’s Setup page on its Clusters tab (/app/os_k8s_search/setup#clusters) and add a cluster:

Names must be unique, and use only lowercase letters, digits, dots, dashes, and underscores, never __. A name cannot be changed after the cluster is created. On the free tier you can register one cluster; more requires a paid license (see Licensing).

A credential is bound to the server it was stored for. Changing a cluster’s API server URL does not carry its stored credentials to the new address: they stop resolving, and the Setup page asks for them again - a different API server is a different trust domain, and the old token is unlikely to be valid there anyway. Credentials stored by Kubernetes Search 1.0.0 carry no binding and keep working; re-save them to bind them. Changing a cluster’s authentication type removes the credentials the previous type used, and a save that supplies only some of the credentials a type needs is refused with a message asking for all of them, or for the stored ones to be deleted first.

Import a kubeconfig

The quickest way to add a cluster is to import a kubeconfig. Upload or paste one, choose a context, and the Setup page fills in the server URL, the CA certificate, and the credential from the selected context - an inline token, or an inline client-certificate-data and client-key-data pair. Kubeconfigs that rely on an exec credential plugin, tokenFile, or certificate/key file paths cannot be imported directly; resolve the credential and paste it inline first (see Requirements). This is convenient for getting started; for a durable setup, prefer a dedicated least-privilege credential as described next.

Authentication

Kubernetes Search authenticates to each cluster with one of these:

Bearer token

A service-account or OIDC token. This is the simplest option and the most common. Store the token on the Setup page; Kubernetes Search keeps it in Splunk’s encrypted credential store (storage/passwords), never in a plain config file.

Give the token the least privilege that covers what your users need. For read-only querying, a service account bound to the built-in view role is a good default. view covers namespaced objects and Namespaces, but none of the cluster-scoped kinds some dashboard panels read - Nodes, PersistentVolumes, StorageClasses, VolumeAttachments, and admission webhook configurations - so the file below also binds k8s-search-cluster-read, a ClusterRole that reads exactly those. Without it, those panels show 403 Forbidden and the panels beside them still load (see Dashboards).

 1apiVersion: v1
 2kind: ServiceAccount
 3metadata:
 4  name: k8s-search-reader
 5  namespace: kube-system
 6---
 7apiVersion: rbac.authorization.k8s.io/v1
 8kind: ClusterRoleBinding
 9metadata:
10  name: k8s-search-reader
11roleRef:
12  apiGroup: rbac.authorization.k8s.io
13  kind: ClusterRole
14  name: view
15subjects:
16  - kind: ServiceAccount
17    name: k8s-search-reader
18    namespace: kube-system
19---
20apiVersion: rbac.authorization.k8s.io/v1
21kind: ClusterRole
22metadata:
23  name: k8s-search-cluster-read
24rules:
25  - apiGroups: [""]
26    resources: ["nodes", "persistentvolumes"]
27    verbs: ["get", "list"]
28  - apiGroups: ["storage.k8s.io"]
29    resources: ["storageclasses", "volumeattachments"]
30    verbs: ["get", "list"]
31  - apiGroups: ["admissionregistration.k8s.io"]
32    resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
33    verbs: ["get", "list"]
34---
35apiVersion: rbac.authorization.k8s.io/v1
36kind: ClusterRoleBinding
37metadata:
38  name: k8s-search-reader-cluster-read
39roleRef:
40  apiGroup: rbac.authorization.k8s.io
41  kind: ClusterRole
42  name: k8s-search-cluster-read
43subjects:
44  - kind: ServiceAccount
45    name: k8s-search-reader
46    namespace: kube-system

Then mint a token for it:

1kubectl -n kube-system create token k8s-search-reader --duration=8760h

The built-in view role does not grant access to Secrets. That is usually what you want for a shared credential. If users need to read Secrets, bind a role that allows it deliberately, and prefer per-user credentials or impersonation (see Access control).

A token stored this way is shared: every user who can search the cluster can read it and use it outside Splunk until you rotate it. To keep it on the search head, have it mint short-lived tokens instead - see Minted tokens.

Client certificate

A client certificate and private key (mutual TLS). Provide both PEMs on the Setup page; they’re stored in the encrypted credential store like tokens.

AWS IAM authentication

Native AWS IAM / EKS authentication is planned for a future release. For now, authenticate to EKS with a bearer token as above.

Minted tokens

A cluster’s stored credential can mint its searches’ tokens instead of being shared with them. Name a service account in Mint tokens for service account (namespace/name; mint_service_account in kubeclusters.conf), and the stored credential is used only inside Splunk, to request a token for that service account through the Kubernetes TokenRequest API. Searches authenticate with that token, which carries the service account’s RBAC and expires ten minutes after it was issued. No user can read the stored credential back through the app, Kubernetes Search administrators included. In Splunk’s credential store the app shares it only with the admin and sc_admin roles, so the users who can read it there are those holding one of them, directly or through a role that imports it, and any role with list_storage_passwords and admin_all_objects, which overrides object permissions. Keep those capabilities to Splunk administrators.

Every user still searches with the same service account, so minting changes nothing about what searches can see. It changes what a user can walk away with: a token taken out of a search dies within minutes and cannot be renewed once that user’s Splunk access is gone. Use it for any cluster whose credential should not be handed to everyone who can search it. Minting works with a bearer token or a client certificate and is available on every license tier, the free one included. It does not apply to an impersonation cluster, whose credential already stays on the search head, or to per-user credentials.

Two identities

Minting needs two identities, and the second must not be able to do what the first does:

• The stored credential - the token or certificate you paste on the Setup page - may create tokens for the search service account: create on serviceaccounts/token, limited to that one name. Nothing else is asked of it.
• The search service account holds the read-only RBAC your searches need, and nothing more. Anything beyond reading would let a token taken out of a search outlive its ten minutes.

Before a token is handed to a search, the app asks the API server, as that token, whether it could outlive its ten minutes: create tokens for itself or another account, read or write Secrets (a service-account token Secret never expires), create pods or workload controllers, reach pods through exec, attach, an ephemeral container or nodes/proxy (a pod keeps a renewed token), or impersonate. Every rule the account holds in its namespace, cluster-wide ones included, must only get, list or watch, and must not read Secrets. A token that fails any of this, or that the API server cannot vouch for, is refused and the search fails with the reason; so is a token issued for longer than the ten minutes requested. The built-in view ClusterRole passes, and so does k8s-search-cluster-read, which only reads.

Two grants the check cannot see: a RoleBinding in another namespace, and - where the authorizer cannot list its rules (GKE, AKS with Azure RBAC) - a write that authorizer grants beyond the ones it is asked about. Keep both out yourself.

The RBAC for a search account splunk-view and a minting account splunk-minter, with k8s-search-cluster-read from Bearer token bound for the cluster-scoped panels:

 1apiVersion: v1
 2kind: ServiceAccount
 3metadata:
 4  name: splunk-view        # what searches run as
 5  namespace: splunk
 6---
 7apiVersion: v1
 8kind: ServiceAccount
 9metadata:
10  name: splunk-minter      # the stored credential
11  namespace: splunk
12---
13apiVersion: rbac.authorization.k8s.io/v1
14kind: ClusterRoleBinding
15metadata:
16  name: splunk-view
17roleRef:
18  apiGroup: rbac.authorization.k8s.io
19  kind: ClusterRole
20  name: view
21subjects:
22  - kind: ServiceAccount
23    name: splunk-view
24    namespace: splunk
25---
26apiVersion: rbac.authorization.k8s.io/v1
27kind: ClusterRoleBinding
28metadata:
29  name: splunk-view-cluster-read
30roleRef:
31  apiGroup: rbac.authorization.k8s.io
32  kind: ClusterRole
33  name: k8s-search-cluster-read
34subjects:
35  - kind: ServiceAccount
36    name: splunk-view
37    namespace: splunk
38---
39apiVersion: rbac.authorization.k8s.io/v1
40kind: Role
41metadata:
42  name: mint-splunk-view
43  namespace: splunk
44rules:
45  - apiGroups: [""]
46    resources: ["serviceaccounts/token"]
47    resourceNames: ["splunk-view"]   # this one account, never a wildcard
48    verbs: ["create"]
49---
50apiVersion: rbac.authorization.k8s.io/v1
51kind: RoleBinding
52metadata:
53  name: mint-splunk-view
54  namespace: splunk
55roleRef:
56  apiGroup: rbac.authorization.k8s.io
57  kind: Role
58  name: mint-splunk-view
59subjects:
60  - kind: ServiceAccount
61    name: splunk-minter
62    namespace: splunk

Then store a token for splunk-minter - or a client certificate for any identity bound to that Role - as the cluster’s credential, with Mint tokens for service account set to splunk/splunk-view. Switching an existing cluster to minting, or back, takes a fresh credential in the same save: a credential that was shared has already been readable by every user who could search the cluster, and one that mints is not safe to share.

A minted token is reused for the same user and cluster until half its lifetime is gone, so a search never starts with less than five minutes left and a dashboard’s panels share one token request. Each token handed to a search is recorded in _internal with the Splunk user it went to and its credential id:

index=_internal "k8s_search audit:" event=token

On Kubernetes 1.30 and later (1.29 with the ServiceAccountTokenJTI feature gate on) the API server’s audit log records the same id on every request the token makes, as authentication.kubernetes.io/credential-id, so a Kubernetes audit event traces back to a Splunk user. See Access control - Auditing usage.

TLS

Kubernetes Search verifies the API server’s certificate against the CA you supply. Two adjustments are available when needed:

• TLS server name - set the expected certificate name when the URL you connect to doesn’t match the certificate’s subject.
• Insecure (skip TLS verify) - skip verification entirely. This is for development against self-signed endpoints only. Don’t use it in production: it exposes the connection to interception, and your credential travels over it.

The API server’s certificate has to meet current Go defaults as well: a certificate with a SHA-1 signature or an RSA key under 2048 bits is refused. Replace such a certificate.

Multiple clusters

Register as many clusters as you need (the free tier allows one) and target them with context= (see the Command reference). Two habits pay off:

• Name for globbing. prod-us-east, prod-eu-west, staging lets a dashboard or search fan out with context=prod-* or context=*.
• Keep names stable. A cluster’s name cannot be changed after it is created, and context= values appear in saved searches and dashboards. A new name means registering the cluster again, with its credentials, and updating every search that names the old one.

Where configuration is stored

The cluster registry (URLs, CA references, default namespaces) lives in the app’s kubeclusters.conf. Credentials never go there - tokens, keys, and CA material are kept in Splunk’s encrypted storage/passwords, private to the app, and referenced by name. In a search head cluster, both the registry and the credentials replicate across members automatically, so a cluster registered on one member is queryable from any other.