Troubleshooting

This page covers where to look when something doesn’t work and the failures people hit most often.

Bulletin messages

Kubernetes Search posts Splunk bulletin messages - the bell in the Splunk bar - for problems that would otherwise surface only when a search fails, and removes each one when its problem ends. They are shown to roles holding edit_k8s_search_clusters or admin_all_objects, and to roles that import one of those; never to search users.

Each message links to the Setup tab that fixes it - License (/app/os_k8s_search/setup#license) for the key and the licensed users, Clusters (/app/os_k8s_search/setup#clusters) for the limits and for a refused credential. The check behind them is the scripted input k8s_search_health.py, which runs every 5 minutes on each search head and logs one line per run to splunkd.log (index=_internal "k8s_search health"). It makes no call to any cluster: a refused credential is reported from what a search saw, so after you update the cluster’s credentials the message clears with the next search the cluster accepts. In a search head cluster, each member reports what its own searches saw.

Each message is posted once. Dismissing it hides it until the problem ends and comes back, or until splunkd restarts; a restart clears every message, and the next check posts what still holds. To stop them, disable the input under Settings - Data inputs - Scripts; the ones already posted last until splunkd restarts.

Where the logs are

Each search writes its own log to the search’s dispatch directory on the search head - the file is k8s_search.log, under $SPLUNK_HOME/var/run/splunk/dispatch/<sid>/. That’s where the detail of what a single command did lives (splunkd cleans these up as the search expires).

The app’s setup handlers, the cache sweeper and the health check log to Splunk’s internal index instead:

index=_internal k8s_search

The cache sweeper and the health check each write one line per run, searchable as index=_internal "k8s_search cache-sweep" and index=_internal "k8s_search health". Changes made on the Setup page are in the app’s audit trail; see Access control - Auditing usage.

To raise the detail while reproducing an issue, set Log level to debug on the Setup page’s Settings tab (level under [logging] in k8s_search.conf), or target a single component (for example the API client) with levels.<component> in the file - see Configuration - logging.

Common issues

Search warnings and errors that send you to the Setup page name the tab that fixes the problem, as a link: add the cluster on the setup page (/app/os_k8s_search/setup#clusters).

A command isn’t recognized, or you get a capability error. Running the commands requires the run_k8s_search capability. Assign the user the k8s_search_user role (or k8s_search_admin). See Access control.

The search can’t reach the cluster. The search head is what connects to the API server, so check that it has a network route to the cluster’s endpoint and that the URL and port are correct. A connection that hangs or times out is almost always network reachability or a firewall, not credentials.

TLS verification fails. A certificate with a SHA-1 signature or an RSA key under 2048 bits is refused whatever the CA, so replace it if the API server presents one. Otherwise, the CA certificate you registered must be the one the API server’s certificate chains to. If the hostname you connect to doesn’t match the certificate, set the server-name override. To confirm it’s a certificate problem (and only in a development environment), temporarily enable the insecure option - if it connects then, the CA or server name is the issue. See Clusters - TLS.

A search fails on an argument that “is not key=value”, an unknown parameter, or a parameter given more than once. Usually an unquoted value with a comma: SPL splits labels=app=web,tier=frontend into two arguments. Quote the whole value - labels="app=web,tier=frontend". See Command reference.

| k8slogs returns _container_error rows. Those containers’ logs could not be read - for example, the credential lacks pods/log in that namespace, or the node is unreachable - while the other containers’ lines came back as usual. The _container_error field says why. A container that has not started yet returns nothing rather than an error.

A search on a minting cluster fails while getting its token. The search service account can do more than read - create tokens, read Secrets, create or exec into pods, or impersonate - or the API server could not confirm what it can do, and the error names the reason. Reduce the account to read-only RBAC (the built-in view ClusterRole passes) and check that the stored credential may create tokens for that account. See Clusters - Minted tokens.

Results are empty. Check the obvious first - the right namespace= and kind=, and whether a labels= or fields= selector is too narrow. If those look right, it’s usually the credential’s RBAC: the identity behind the cluster can’t list that resource. For example, the built-in view role can’t read Secrets. Confirm with a kind you know is readable, like | k8s kind=namespaces.

A second cluster is refused. The free tier allows one cluster. Registering more requires a paid license. See Licensing.

A search is refused because the free tier is licensed for one user. The free tier serves one user, the first to run a search, and the message names them. An administrator can change the user on the Setup page’s License tab, or a paid license admits your whole team. See Licensing.

The app won’t run in a search head cluster. Running in an SHC requires a paid license.

The bell says a cluster refused its credentials. The cluster answered 401 to its stored credential on the latest search that used it: the token expired or was revoked, the certificate expired, or the credential belongs to another cluster. Store a working one on the Clusters tab; the message clears with the next search the cluster accepts.

Credentials stop resolving after you change a cluster’s server. A credential is bound to the server it was stored for, so re-pointing a cluster does not carry its credentials along. Enter them again on the Setup page; see Clusters.

Changing a cluster’s impersonation setting is refused. The cluster still stores its default credentials. Delete them first, change the setting, then store the credential the new mode needs; see Access control.

An impersonation query is refused with a message asking to retry. The search head has hit its cap on server-side impersonation queries running at once, and refused the query rather than fork without limit. Retry; a search head that hits the cap often is serving more concurrent impersonation queries than it is sized for.

Impersonation searches fail from a scheduled search. A scheduled or saved search runs as nobody, with no user identity to impersonate, so it’s refused. Impersonation is for interactive analyst queries. See Access control.

Data looks stale. Results may be served from the short-lived cache. Add cache=0 to force a live call, and check the _cache_hit and _cache_age_seconds fields on the rows to see what you got. See Concepts - caching.

One cluster fails during a fan-out. When you target several clusters with context=*, a cluster that can’t be reached emits a single error row and the others still return. Look for the error row to see which cluster failed and why.

Verify a connection

The quickest end-to-end check of a registration is to list a cluster-scoped kind that any read credential can see:

| k8s kind=namespaces context=production

If that returns rows, the URL, CA, credential, and network path are all good, and any empty result from a more specific query is about the query or RBAC, not the connection.