Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk
Kubernetes Search in Splunk Monitoring Docker in Splunk Monitoring Kubernetes in Splunk Monitoring OpenShift in Splunk Monitoring Linux in Splunk Monitoring Windows Containers in Splunk Forwarding Logs to Elasticsearch and OpenSearch Forwarding Logs to QRadar
Documentation Blog Contact Us Pricing Evaluation licenses Partners License Agreement Privacy Policy Refund & Cancellation Acceptable Use Policy FAQ
Kubernetes Search

Troubleshooting

This page covers where to look when something doesn’t work and the failures people hit most often.

Where the logs are

Each search writes its own log to the search’s dispatch directory on the search head - the file is k8s_search.log, under $SPLUNK_HOME/var/run/splunk/dispatch/<sid>/. That’s where the detail of what a single command did lives (splunkd cleans these up as the search expires).

The app’s setup handlers and the cache sweeper log to Splunk’s internal index instead:

index=_internal k8s_search

To raise the detail while reproducing an issue, set level = debug under [logging] in k8s_search.conf, or target a single component (for example the API client) with levels.<component> - see Configuration - logging.

Common issues

A command isn’t recognized, or you get a capability error. Running the commands requires the run_k8s_search capability. Assign the user the role_k8s_search_user role (or role_k8s_search_admin). See Access control.

The search can’t reach the cluster. The search head is what connects to the API server, so check that it has a network route to the cluster’s endpoint and that the URL and port are correct. A connection that hangs or times out is almost always network reachability or a firewall, not credentials.

TLS verification fails. The CA certificate you registered must be the one the API server’s certificate chains to. If the hostname you connect to doesn’t match the certificate, set the server-name override. To confirm it’s a certificate problem (and only in a development environment), temporarily enable the insecure option - if it connects then, the CA or server name is the issue. See Clusters - TLS.

Results are empty. Check the obvious first - the right namespace= and kind=, and whether a labels= or fields= selector is too narrow. If those look right, it’s usually the credential’s RBAC: the identity behind the cluster can’t list that resource. For example, the built-in view role can’t read Secrets. Confirm with a kind you know is readable, like | k8s kind=namespaces.

A second cluster is refused. The free tier allows one cluster. Registering more requires a paid license. See Licensing.

The app won’t run in a search head cluster. Running in an SHC requires a paid license.

Impersonation searches fail from a scheduled search. A scheduled or saved search runs as nobody, with no user identity to impersonate, so it’s refused. Impersonation is for interactive analyst queries. See Access control.

Data looks stale. Results may be served from the short-lived cache. Add cache=0 to force a live call, and check the _cache_hit and _cache_age_seconds fields on the rows to see what you got. See Concepts - caching.

One cluster fails during a fan-out. When you target several clusters with context=*, a cluster that can’t be reached emits a single error row and the others still return. Look for the error row to see which cluster failed and why.

Verify a connection

The quickest end-to-end check of a registration is to list a cluster-scoped kind that any read credential can see:

| k8s kind=namespaces context=production

If that returns rows, the URL, CA, credential, and network path are all good, and any empty result from a more specific query is about the query or RBAC, not the connection.