Configuration - Kubernetes Search

Most of Kubernetes Search is configured on the Setup page, clusters and credentials on their own tabs (see Clusters). This page covers the rest: runtime values you rarely need to touch, but that are there when you want to tune request timeouts, fan-out concurrency, the cache, or log verbosity. Most of them are on the Setup page as well:

The Settings tab (/app/os_k8s_search/setup#settings) sets the Kubernetes API and log request timeouts, the fan-out concurrency, the longest log line | k8slogs reads, the splunkd request timeout, and the search log level.
The Cache tab (/app/os_k8s_search/setup#cache) turns the cache on or off and clears it.
The License tab holds the license key; see Licensing.

A field left blank on the Settings tab keeps the shipped default, and clearing a field is how you reset it. The tab checks each value before saving and refuses one the search commands could not read, with a message next to the field. On Splunk Cloud, where nobody can edit the app’s files, the Setup page is how you set them.

Everything they save lives in k8s_search.conf, under local/, and in a search head cluster it replicates to every member. A few keys stay in the file only: the cache directory ([cache] dir), the log file and per-component levels ([logging] file and levels.<component>), and how splunkd’s certificate is checked (the [splunk] tls_* keys). On a self-managed search head you can edit the file directly - the copy under local/, never the shipped default/:

$SPLUNK_HOME/etc/apps/os_k8s_search/local/k8s_search.conf

An app upgrade overwrites default/, so a value set there is lost on the next release; local/ always wins and survives upgrades. Set only the keys you want to change - anything you leave out keeps its default. Each search reads the file when it runs, so a change takes effect on your next search, with no restart.

When you edit the file by hand, a boolean takes Splunk’s spellings in any letter case - true, t, yes, y, on, 1 and false, f, no, n, off, 0. A value the search commands cannot parse - a duration without its unit, a size that is not a whole number - fails every search until it is fixed, so edit with care.

The bundled README/k8s_search.conf.spec is the exhaustive schema; this page covers the settings worth tuning in practice.

Kubernetes API requests

The Settings tab, and [kubernetes] in the file: how the commands talk to your cluster API servers.

Setting	Default	What it does
`api_request_timeout`	`30s`	Caps a single non-streaming API call (discover, list, get) from `\| k8s`, `\| k8sevents`, and `\| k8sdescribe`. A wall-clock bound from the search side, so a hung connection can’t leave a search stuck.
`logs_api_request_timeout`	`30s`	The same bound for `\| k8slogs`. A value of `0` means “use the default”, not “unbounded”. Raise it only for a large `tail=` or `since=` read.
`fan_out_concurrency`	`8`	How many clusters one search queries at once during a fan-out (`context=*` or a glob), and how many container logs `\| k8slogs` reads at once per cluster. A search’s `concurrency=` argument overrides it for the cluster fan-out, up to 128. Raise it for very wide fleets; lower it to keep CPU off a small search head.
`max_log_line_bytes`	`1048576`	The largest single container log line `\| k8slogs` returns, in bytes (1 MiB). Raise it if you see a “token too long” error in the search log - usually a one-line JVM stack trace or an init-container bootstrap.

The Settings tab takes the two timeouts as a duration with its unit (30s, 1m), above zero and at most 24h; the fan-out concurrency from 1 to 128; and the log line size from 1024 to 67108864 bytes (64 MiB).

local/k8s_search.conf ini

1[kubernetes]
2fan_out_concurrency = 16
3api_request_timeout = 1m

Cache

[cache] tunes the on-disk result cache. The Cache tab turns it on or off and clears it; the time-to-live values and size caps are set in the file, or through the app’s settings REST endpoint, which takes every key below except dir. The defaults suit most installs; see Concepts - caching for how the cache behaves and the per-search cache= override.

Setting	Default	What it does
`enabled`	`true`	Master on/off switch. `false` runs every search uncached. The Cache tab toggles this too.
`dir`	`$SPLUNK_HOME/var/run/os_k8s_search/cache`	Where cached responses live on the search head. File only.
`discovery_ttl`	`10m`	How long the cluster’s API discovery (its groups and kinds) is cached. Discovery is schema, not state, so a long TTL is safe.
`list_default_ttl`	`30s`	Default freshness for list queries (`\| k8s kind=pods`, `\| k8sevents`) when a search doesn’t set `cache=`.
`get_default_ttl`	`10s`	Default freshness for single-object gets (`\| k8s kind=pods/name`, `\| k8sdescribe`).
`max_size_bytes`	`268435456`	Soft cap on total cache size, in bytes (256 MiB). The oldest entries are evicted when it is exceeded.
`max_entries`	`4096`	Soft cap on the number of cached entries, enforced the same way.

To flush the cache immediately - for example right after rotating a credential, rather than waiting out the TTL - use the Clear cache button on the Setup page. On a Splunk Cloud search head where var/run/ isn’t writable, turn the cache off on the Cache tab (enabled = false); the app also falls back to running uncached on its own if it can’t write.

Logging

[logging] controls the search-command log, the per-search log you read when diagnosing a problem (see Troubleshooting). The Settings tab sets level to debug, info, warn or error; the other values and keys are set in the file.

Setting	Default	What it does
`level`	`info`	Master log level: `trace`, `debug`, `info`, `warn`, `error`, or `fatal`. Raise it to `debug` while you reproduce an issue; the change takes effect on your next search.
`file`	`k8s_search.log`	Log file. A relative path lands in the search’s dispatch directory, which Splunk cleans up; an absolute path goes where you point it and is yours to rotate.
`levels.<component>`	-	Raise the level for one component without flooding the rest. For example, `levels.core/kubernetes/api = debug` traces just the API calls, and `levels.core/httpx = debug` the HTTP transport and TLS. The full component list is in `README/k8s_search.conf.spec`.

local/k8s_search.conf ini

1[logging]
2level = info
3levels.core/kubernetes/api = debug

At trace, the HTTP client writes request dumps and response bodies into the search log - the file you send to support. Set trace for one component, and only for as long as you need it.

Connection to splunkd

[splunk] governs how the app reaches the local splunkd REST API to resolve the credentials a search needs - not how it reaches your Kubernetes clusters (that is on the Clusters page). The defaults match a standard Splunk install, and you won’t normally change them. The Settings tab sets request_timeout; the tls_* keys are set in the file only.

Setting	Default	What it does
`request_timeout`	`30s`	Caps a single REST call to the local splunkd.
`tls_server_name`	(empty)	The hostname the splunkd certificate must match. Set it only when you have replaced splunkd’s certificate with one that carries a proper DNS SAN.
`tls_insecure`	`false`	Skip certificate verification for that local call. For an install with a custom certificate the app cannot otherwise verify; the call is still authenticated by the Splunk session.

The remaining keys (tls_ca_file, tls_cert_file, tls_server_name_from_cert) point at the splunkd CA and certificate and pin the connection on loopback. Leave them at their defaults on a stock Splunk install; README/k8s_search.conf.spec documents them in full.

License

[license] holds your license key.

Setting	Default	What it does
`key`	(empty)	The signed key from your order confirmation. Empty is the free tier. Prefer entering it on the Setup page’s License tab, which writes it here for you.
`seat`	(empty)	The free tier’s one user. Empty until the first person to run a search takes it; change it on the License tab. A license ignores it.

See Licensing for what a key unlocks.