Kubernetes Search vs Monitoring and OpenTelemetry

There are three common ways to get Kubernetes visibility into your tooling, and they answer different questions. This page explains the distinction so you can pick the right one - or, more often, combine them.

The core distinction: stored vs live

Most observability works by ingestion: something collects telemetry from the cluster and ships it to a backend, where it’s stored and you query the stored copy. Monitoring Kubernetes and OpenTelemetry are both ingestion approaches. Ingestion is what gives you history - trends over weeks, dashboards over time, and alerts that fire on stored data - at the cost of storage and, in Splunk, license volume.

Kubernetes Search is the opposite. It stores nothing. It reads the live Kubernetes API on demand and streams the answer into your search. That gives you the current truth with no storage cost, but no history of its own.

So the first question is always: do you need what has been happening over time (ingest it) or what is true in the cluster right now (search it live)? Most teams need both.

Monitoring Kubernetes

What it is. Outcold Solutions’ turnkey pipeline: the Collectord agent runs in your cluster (a DaemonSet plus a small add-on) and forwards container logs, host logs, metadata-enriched metrics, audit logs, and Kubernetes events and objects into Splunk, with dashboards and alerts on top.

How it works. The agent collects continuously and sends to Splunk’s HTTP Event Collector. The data lands in indexes you own and search like any other Splunk data.

Strengths. History and retention; alerting on logs and metrics; correlation with the rest of your Splunk data; a maintained set of dashboards and alerts. It is the system of record for what your workloads logged and how they performed.

Costs. Storage and Splunk license volume for the ingested data, and an agent to run in every cluster.

Use it for long-term observability: investigating an outage after the fact, capacity trends, audit retention, and alerting.

OpenTelemetry

What it is. The open, vendor-neutral standard for telemetry - traces, metrics, and logs - from the CNCF. You instrument applications with the OpenTelemetry SDKs and/or run the OpenTelemetry Collector, then export to any backend that speaks OTLP, Splunk included.

How it works. Like Monitoring Kubernetes, it ingests: the Collector receives and exports telemetry to a backend that stores it. Its distinctive strength is distributed tracing and standardized, portable instrumentation that isn’t tied to one vendor.

Strengths. Application-level instrumentation and traces; a single instrumentation standard across many backends; portability if you change vendors.

Costs. Instrumentation effort, a Collector to operate, and storage in whatever backend you choose. It is a framework you assemble, not a finished Kubernetes app.

Use it for application performance and tracing, and when vendor-neutral instrumentation is a requirement.

OpenTelemetry and Monitoring Kubernetes aren’t mutually exclusive - many teams send OTLP application telemetry through the Collector and use Monitoring Kubernetes for the cluster’s logs, metrics, and objects. Both are ingestion, both give you history.

Kubernetes Search

What it is. Live, on-demand access to the Kubernetes API from the Splunk search bar. No agent in the cluster, no ingestion, no index. Like kubectl, but inside Splunk, RBAC-controlled, and across every cluster at once.

How it works. A search command on the search head calls the API server directly and returns the current state. See Concepts.

Strengths. Always current; zero storage and zero license volume; nothing to deploy in the cluster; multi-cluster from one pane; and controlled, audited read access for analysts who shouldn’t have kubectl or cluster credentials. It pairs naturally with ingested data in the same Splunk session.

Costs and limits. No history of its own (you see now, bounded by what Kubernetes itself retains); logs are a snapshot, not a live tail; read-only.

Use it for live investigation, confirming current state, ad-hoc multi-cluster queries, and giving people read access to clusters without distributing credentials.

Side by side

On that Alerting row: Kubernetes Search alerts by scheduling a saved search over the live commands, so each run evaluates current cluster state - ideal for “X is wrong right now.” What it can’t do is alert on historical windows or trends over time; that’s ingestion’s job. See Use cases for examples.

Use them together

The approaches are complementary, and the strongest setup uses ingestion for history and Kubernetes Search for the live picture - in the same Splunk session.

A typical incident flow: you alert and triage on ingested data (Monitoring Kubernetes shows the error spike and the OOM kills over the last hour), identify the offending workload, then switch to Kubernetes Search to see its state right now - is the pod still crash-looping, what do its current events say, is it scheduled, what’s the node’s pressure - without leaving Splunk for a terminal and without cluster credentials.

index=kubernetes namespace=payments "OOMKilled" earliest=-1h
| stats count by pod_name

| k8sdescribe kind=pod/payments-api-7d9f8 namespace=payments

History tells you what happened; the live query tells you whether it’s still happening. Run Monitoring Kubernetes (or your OpenTelemetry pipeline) for the first, and Kubernetes Search for the second.