Installation - Kubernetes Search

Install on a search head, assign roles, register a cluster, run your first search, upgrade

This guide takes you from zero to your first live query: install the app on a search head, assign the roles, register a cluster, and run | k8s. A typical setup takes under 10 minutes, and the free tier needs no license key. If you want more than one user, more than one cluster, or per-user access, contact sales@outcoldsolutions.com to request a 30-day evaluation.

Requirements

In brief: Splunk Enterprise 9.4, 10.0, 10.2, 10.4, or 10.6 on a search head, or Splunk Cloud, a network route from that search head to each cluster’s API server over HTTPS, Kubernetes 1.20 or later, and a credential per cluster (a bearer token or client certificate). The full compatibility matrix - supported search-head platforms and Kubernetes versions - is in Requirements; obtaining a least-privilege credential is covered in Clusters.

Install the app

Install the latest Kubernetes Search app from Splunkbase on your search heads only. In a search head cluster, push it through the deployer the same way you deploy any app. Running inside a search head cluster requires a paid license - see Licensing.

The app installs two custom roles and several search commands. No restart of your indexing tier is needed.

Assign roles

Kubernetes Search separates the people who configure clusters from the people who query them. Two roles ship with the app:

k8s_search_user - run the search commands (| k8s, | k8slogs, | k8sevents, | k8sdescribe) and manage your own per-user credentials. Grant this to anyone who should query clusters.
k8s_search_admin - everything a user can do, plus register, edit, and delete cluster definitions and manage every cluster’s credentials. Grant this to whoever owns the connections.

A full Splunk administrator (the built-in admin_all_objects capability) can do everything without being assigned either role. Assign roles under Settings - Roles. The capabilities behind them are covered in Access control.

Register your first cluster

Open the app and go to the Setup page (or Manage Apps - Kubernetes Search - Set up). The Setup page has five tabs - Clusters, Credentials (each user’s own per-user credentials), Cache, Settings and License - and each opens by link: /app/os_k8s_search/setup#clusters opens Clusters, and likewise #credentials, #cache, #settings and #license. On the Clusters tab, add a cluster with a name, the API server URL, its CA certificate, and a credential:

The fastest path is to import a kubeconfig: paste or upload one and the setup page reads the API URL, the CA, and a token from the context you pick. The full reference - including client certificates, default namespaces, per-user credentials, and minting short-lived tokens so the stored credential never reaches a search - is in Clusters. Request timeouts, fan-out concurrency and the search log level are on the Settings tab; see Configuration.

On the free tier you can register one cluster. Registering a second requires a paid license. The first cluster you add keeps working regardless.

Run your first search

Open Search in the app and list the pods in a namespace:

| k8s kind=pods namespace=payments

Each row is one Kubernetes object: _raw holds the full JSON, with kind, name, and namespace pulled out as fields. Nothing was written to an index - the command queried the API live and streamed the result into your search. From here you can pipe into any SPL you already know:

| k8s kind=pods namespace=payments
| spath path=status.phase output=phase
| stats count by phase

Try the other commands - stream logs, read events, or describe an object and its events together:

| k8slogs namespace=payments pods=payments-api-* tail=200
| k8sevents namespace=payments type=Warning
| k8sdescribe kind=pod/payments-api-7d9f8 namespace=payments

If you registered more than one cluster, add context= to pick one or fan out across many in parallel:

| k8s kind=nodes context=*
| stats count by cluster

The full syntax for every command is in the Command reference.

Free tier and licensing

Kubernetes Search runs with no license key on the free tier: one user - the first to run a search - one cluster, a single standalone search head, and a shared credential or minted tokens. A paid license, priced per user, unlocks your whole team, any number of clusters, search head clustering, per-user credentials, and impersonation. See Licensing for the full breakdown and how to enter a key.

Search head clustering

In a search head cluster, deploy the app through the deployer. The app’s configuration - registered clusters, credentials, and settings - replicates across members automatically, so a cluster you register on one member is visible to a search that lands on any other. Running in an SHC requires a paid license. See Licensing and the credential notes in Access control.

Upgrade

Install the new version over the old one the same way you installed it - from Splunkbase, or through the deployer in a search head cluster - then restart splunkd. Registered clusters, credentials, and the settings under local/ carry over. Check Release history for what changed.

From 1.0.0 to 1.1.0

A few changes in 1.1.0 need action from you:

Restart splunkd after upgrading, so the Setup page’s new permissions apply. 1.1.0 also adds a scripted input, k8s_search_health.py, which posts bulletin messages every 5 minutes; disable it to turn them off.
Quote comma values. SPL splits labels=app,env into two arguments, and the commands now fail such a search instead of silently running the broader selector; a parameter given twice is refused too. Check saved searches, alerts, and your own dashboards, and quote the whole value: labels="app=web,tier=frontend". See Command reference.
Move an override of the cache sweeper. Its input is renamed from [script://./bin/cache_sweep.py] to [script://./bin/k8s_search_cache_sweep.py]. An override of the old stanza in local/inputs.conf - a disabled toggle or a custom interval - no longer applies, and the renamed sweeper starts with its defaults (enabled, every 300 seconds): move the override to the new name. An in-place upgrade leaves the old bin/cache_sweep.py on disk, so an override that keeps the old stanza enabled keeps that script running beside the new sweeper; remove the old override rather than keeping both.
Delete a cluster’s default credentials before turning impersonation on or off. The change is refused while they are stored. See Access control.
Re-save each cluster’s credentials to bind them to its API server. Credentials stored by 1.0.0 keep working unbound; once bound, a credential stops resolving if the cluster’s server changes. See Clusters.
Replace an API server certificate with a SHA-1 signature or an RSA key under 2048 bits. 1.1.0 refuses it.

A cluster that shares its credential with every user who can search it can now keep that credential on the search head instead, by minting short-lived tokens; see Clusters - Minted tokens.