Cutting Splunk ingestion costs from Kubernetes: a practical playbook

Splunk bills by what you ingest, and a Kubernetes cluster is an ingestion machine: every container’s stdout, kubelet probing /healthz around the clock, sidecars chattering at debug level, and application logs that repeat a timestamp Splunk already stores. Most of those bytes are never searched by anyone. They just accrue license usage and storage until the bill forces the question: what are we actually paying to keep?

The wrong answer is asking every team to log less - that campaign never ends and never sticks. The right place to cut is the collection edge, where one annotation on a workload decides what leaves the node at all. This playbook walks seven levers in the order you should pull them, each verified on a live cluster with before/after numbers from Splunk itself.

Everything below uses Collectord annotations on Kubernetes; the same annotations work identically on OpenShift (oc instead of kubectl, and openshift_* instead of kubernetes_* in field names, index names, and sourcetypes). The demo numbers were measured on an OpenShift cluster - the mechanics are the same binary on both.

Measure first: where the money goes

Cutting blind is how you end up dropping the audit trail while the debug firehose keeps running. Two searches tell you where the volume actually comes from.

License usage by index and sourcetype - the numbers Splunk actually bills on:

index=_internal source=*license_usage.log* type=Usage
| stats sum(b) as bytes by idx, st
| eval MB=round(bytes/1024/1024,1)
| sort -bytes

This is coarse but authoritative. On our demo cluster it immediately reframed the problem: container logs were not the biggest line item at all -

1idx        st                        MB
2openshift  openshift_host_logs    193.7
3openshift  openshift_prometheus    34.6
4openshift  openshift_logs          26.0
5openshift  openshift_events        13.7

The node’s host logs were seven times the container logs, and drilling in showed why: the Kubernetes API server audit log alone was producing over 100,000 events per hour on an otherwise idle cluster. Lever 7 exists because of exactly this pattern.

License usage cannot tell you which namespace or which container is expensive, though - for that, approximate bytes from the events themselves:

index=kubernetes sourcetype=kubernetes_logs earliest=-24h
| eval bytes=len(_raw)
| stats sum(bytes) as bytes, count by kubernetes_namespace, kubernetes_container_name
| eval MB=round(bytes/1024/1024,1)
| sort -bytes

Run it over a representative day and you have your target list, ranked. The Monitoring Kubernetes and Monitoring OpenShift apps also ship a Splunk Usage dashboard under Setup that charts this continuously.

The demo: one namespace that does everything wrong

To put numbers on each lever, the demo cluster runs a webportal namespace with the noise patterns you will recognize from your own target list:

• a web container writing access logs, where three out of four lines are kubelet probes hitting /healthz and /readyz;
• an app container writing Java-style logs - two thirds DEBUG, and every line prefixed with a timestamp, an instance name, and a thread name;
• an envoy sidecar producing debug output nobody has ever searched;
• a separate clickstream deployment emitting five telemetry events per second across a rotating population of 200 users.

Baseline, measured over seven minutes in Splunk with sum(len(_raw)):

1container     events/min   KB/min
2web                  239     22.3
3app                  180     22.8
4envoy                179     16.6
5clickstream          297     19.4
6total                895     81.1

Every change below is an annotation on one of these two deployments. No application changes, no ConfigMap edits, no Collectord restart.

Lever 1: stop collecting what nobody reads

The cheapest log line is the one that never leaves the node. For the envoy sidecar - debug output, fully redundant with the metrics we already collect - the whole stream goes:

1spec:
2  template:
3    metadata:
4      annotations:
5        collectord.io/envoy--logs-output: 'devnull'

The envoy-- prefix scopes the annotation to that one container; the other containers in the pod are untouched. With devnull, Collectord still reads the log file and advances its position tracker - it just acknowledges the events instead of forwarding them. That detail matters when you change your mind: flip the annotation back to splunk and forwarding resumes from the moment of the switch, with no replay of the muted backlog.

The alternative is collectord.io/envoy--logs-disabled: 'true', which stops reading the file entirely. It saves the little CPU that reading costs, but the position tracker freezes - re-enable it later and Collectord replays everything still on disk since the freeze. Pick devnull to mute a container you might unmute; pick disabled for a container whose logs you want the option to backfill later. The layering post walks through the same distinction from the annotation-precedence side.

Two bigger versions of the same lever:

• Flip the default. Some clusters opt out of log forwarding instead of in: start Collectord with --env "COLLECTOR__LOGS_OUTPUT=input.files__output=devnull", then opt workloads in with collectord.io/logs-output: 'splunk' on the namespaces that want logs. This is the cleanest endgame for clusters where only a few teams ever look at Splunk.
• Per-datatype outputs. Logs are not the only stream: collectord.io/stats-output, procstats-output, netstats-output, and nettable-output accept devnull too, so a namespace can keep logs and drop its process and network metrics, or the reverse.

Lever 2: drop the lines that are pure noise

Inside a stream you want to keep, some lines are still pure cost. The classic offenders are health-check requests - kubelet probes /healthz and /readyz on every container, every few seconds, forever - and DEBUG output that ships to production because nobody flipped the level back.

There is no blacklist annotation for container logs; a replace pipe with an empty replacement is the drop mechanism. On the web container, drop successful probe requests:

1collectord.io/web--logs-replace.1-search: '^\S+ - - \[[^\]]+\] "GET /(healthz|readyz)[^"]*" 200 .+$'
2collectord.io/web--logs-replace.1-val: ''

On the app container, drop debug lines:

1collectord.io/app--logs-replace.1-search: '^.+ DEBUG .+$'
2collectord.io/app--logs-replace.1-val: ''

An event whose message ends up empty is never sent to Splunk. Anchor the patterns (^...$) so a line mentioning the word DEBUG mid-sentence doesn’t vanish, and remember that kubectl logs still shows everything - you are choosing what to pay to index, not what the container writes.

When the lines you want to keep are the short list - an audit sidecar where only AUDIT records matter - invert the logic with collectord.io/logs-whitelist and forward only matching lines. We covered whitelist in depth in the PII masking post, where it doubles as a compliance control.

Lever 3: stop paying to ingest the same timestamp twice

Look at one line from the app container:

12026-08-23 03:29:10.123 [webportal-6f8c] [http-nio-8080-exec-3] INFO c.o.w.CheckoutService - order submitted orderId=ord-146 total=42.10

The first 64 characters are a timestamp Splunk will store anyway as _time, an instance name Collectord already attaches as pod metadata, and a thread name nobody searches. That prefix is 47% of the line, on every line, forever. Field extraction removes it at the edge:

1collectord.io/app--logs-extraction: '^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) \[[^\]]+\] \[[^\]]+\] (.+)$'
2collectord.io/app--logs-timestampfield: 'ts'
3collectord.io/app--logs-timestampformat: '2006-01-02 15:04:05.000'

The unnamed capture group (.+) becomes the event message - _raw in Splunk is now just INFO c.o.w.CheckoutService - order submitted orderId=ord-146 total=42.10. The ts group feeds _time, so the timestamp is not lost, it has simply stopped being paid for twice. As a bonus, _time now reflects when the application logged the line rather than when the runtime wrote it.

Three details to get right:

• Anchor the regex and keep exactly one match per event. A pattern that matches zero times - or more than once inside a joined multiline event - leaves the message untouched and tags the event with collectord_errors=extract, which is also how you find your misses: index=kubernetes collectord_errors=extract.
• If the regex has several unnamed groups, the last one becomes the message. Use a single unnamed group, or name the message group and select it with logs-extractionMessageField.
• Timestamp formats use Go’s reference-date layout (2006-01-02 15:04:05.000), and @unixtimestamp for epoch values.

This idea - strip the timestamp, keep the signal - is old advice with new numbers: we measured timestamps alone at 17% of container log volume back in 2019. Verbose framework prefixes push it much higher.

Lever 4: sample the streams where trends are enough

The clickstream deployment emits five events per second, and every consumer of that data looks at rates and distributions, never individual lines. Trend data tolerates sampling:

1collectord.io/logs-sampling-percent: '20'

That alone keeps a random ~20% of lines - fine for dashboards, fatal for investigations, because you now have one in five of any given user’s events. Hash-based sampling fixes the fatal part. Give Collectord a key, and it keeps or drops all events sharing that key value:

1collectord.io/logs-sampling-percent: '20'
2collectord.io/logs-sampling-key: 'user=(?P<key>[0-9a-f]+)'

Collectord hashes the captured key and keeps the event if the hash falls in the kept bucket - deterministically, across restarts and across nodes. Measured on the demo: before sampling, the stream carried all 200 users; after, 47 users remained (23.5% of them) - and for each surviving user, every one of their events was still there, at the same per-user rate as before (10.4 events per user per window, before and after). You cut the volume to roughly a quarter and can still reconstruct any kept user’s complete session.

Worth knowing before you rely on it:

• Lines where the key regex does not match are forwarded at 100%, not sampled. That is a feature - your error lines without a user= field survive sampling - but it means a too-narrow key regex quietly disables sampling for everything else.
• The kept share is approximate, and for keyed sampling it converges on the target only when the key has real cardinality and variety. Highly regular keys can cluster in hash space - our first attempt used twenty sequential IDs (u00-u19) and the 20% sample kept none of them. After enabling sampling, verify the kept share with a dc() search before trusting it; never build billing on “exactly 20%”.
• Sampling is the wrong tool whenever single events carry obligations: audit trails, security events, payment records. Drop or route those deliberately, never probabilistically.

Lever 5: cap the blast radius

The levers so far assume you know which streams are noisy. The next outage will invent a new one - a crash loop stack-tracing at full speed, a debug flag left on in a hot code path. A throughput cap turns “runaway container floods the license” into a bounded problem:

1collectord.io/logs-ThruputPerSecond: 128Kb

The cap meters message bytes per second for that container. When a burst exceeds it, Collectord does not discard the excess - it stops reading and lets the backlog sit in the log file on disk, resuming as budget frees up, and logs a pipeline is getting throttled warning at most once a minute. A sustained overflow eventually loses data anyway - the kubelet rotates container logs, and what rotates away before Collectord catches up is gone - so treat the cap as a circuit breaker for abnormal behavior, not a steady-state sampling mechanism. The same knob exists per instance as thruputPerSecond in the [general] section of the ConfigMap, and per input for host and application logs.

Its companion guard rails bound the time dimension the way the cap bounds rate:

1collectord.io/logs-TooOldEvents: '168h'
2collectord.io/logs-TooNewEvents: '1h'

Events whose own timestamp falls outside the window around now are dropped and acknowledged. This is what saves you when Collectord is first installed on a node with months of log history on disk, or when a container you re-enable after logs-disabled offers to replay its life story.

Lever 6: route what you keep to cheaper shelves

Everything so far reduces what you ingest. Routing does not - a byte costs the same license whichever index it lands in - but it controls the other cost curve: storage and retention. Splunk retention is per index, so splitting data by value lets each class expire on its own schedule instead of everything living as long as the most-regulated event.

The workhorse is namespace-level index routing, one annotation per team:

1kubectl annotate namespace webportal collectord.io/logs-index=kubernetes_apps_shortlived

For splitting within a stream, override pipes re-route individual lines by regex - send the access-log portion of a container to a 7-day index while its error lines keep 90 days:

1collectord.io/logs-override.1-match: '"GET /'
2collectord.io/logs-override.1-index: 'kubernetes_access_7d'

Two cautions. First, the HEC token must include any index you route to in its allowed-indexes list - otherwise Splunk rejects the write, and Collectord’s shipped default (incorrectIndexBehavior = RedirectToDefault) quietly re-sends those events to the token’s default index, undoing your routing. Second, watch multi-output annotations (logs-output: 'splunk::apps[...],splunk::security[...]') - fan-out is a feature for compliance, but every listed destination ingests a full copy, which multiplies license usage rather than reducing it.

Lever 7: look beyond container logs

Container logs get all the attention, but the measurement section above showed host logs outweighing them seven to one on our cluster - dominated by the API server audit log. The same edge-filtering exists for those inputs, in the ConfigMap rather than annotations:

• Journald: [input.journald] supports numbered whitelist.N / blacklist.N regexes. The shipped OpenShift config enables one (blacklist.0 = ^I\d+.*$) that drops verbose INFO messages from cluster components; the Kubernetes template ships the key commented out, so set the same blacklist.0 yourself if kubelet and friends dominate your host logs.
• File groups like the audit log input ([input.files::audit-logs] on OpenShift): the same whitelist.N / blacklist.N keys apply, so you can keep the audit records that matter to you (writes, denials) and drop the reads-of-nothing that make up most of the volume. Tuning the cluster’s audit policy upstream helps even more.
• Metrics: Prometheus inputs accept per-metric whitelist.N regexes, which is the difference between “all of etcd’s metrics” and “the twenty you alert on” - review the shipped filters in each [input.prometheus::*] section against what your dashboards actually query. Per pod or namespace, the stats-output / netstats-output / nettable-output annotations from lever 1 switch whole metric datatypes off. Socket-row grouping (group = true under [input.net_socket_table]) cuts that sourcetype several-fold but already ships enabled in current configs - verify it only on installs that predate it or were customized.

The results

After applying levers 1-4 to the demo namespace (envoy to devnull, probe and DEBUG lines dropped, the Java prefix extracted away, clickstream sampled at 20% with a user key), the same seven-minute measurement:

1container     events/min      KB/min
2web           239 ->  60      22.3 ->  5.6
3app           180 ->  60      22.8 ->  4.2
4envoy         179 ->   0      16.6 ->  0
5clickstream   297 ->  70      19.4 ->  4.6
6total         895 -> 190      81.1 -> 14.4

82% of the byte volume gone, and nothing anyone searches was lost: real requests still land in full, INFO and above still land (smaller), lines without a user key still land at 100%, and any kept user’s telemetry is complete. Your percentage will differ - that is the point of measuring first - but the shape repeats everywhere: most Kubernetes log volume is probes, debug, redundant prefixes, and telemetry that tolerates sampling.

For scale: this one small namespace went from 3.6 GB to 0.6 GB per month. Multiply by the hundreds of workloads in a production cluster and the license line item moves visibly.

Keep it cut: enforcement and verification

A cost cut that lives in tribal knowledge regresses in a quarter. Make it structural:

• Namespace annotations make the routing and sampling defaults owned by the team, inherited by every new workload automatically.
• A Configuration CRD applies rules by metadata regex across namespaces - drop probes everywhere, cap throughput on everything in *-dev - and with force: true the platform team’s rule beats workload-level overrides. The layering post covers where each rule should live.
• Verify what is actually applied with collectord describe - it prints the merged annotation set for a pod with each value’s origin - and watch index=kubernetes collectord_errors=* for pipes that are misfiring instead of saving you money.
• Re-run the measurement searches monthly. New workloads arrive noisy; the target list changes.

Collectord also reports on itself. Its internal metrics arrive under source=collectord:* in the Prometheus sourcetype, and the pipe gauge is a live inventory of the transform rules this playbook installs - replace, whitelist, extraction, sampling, override, throughput. Every Collectord instance publishes its own series (each DaemonSet pod, plus the addon with an :addon suffix on the source), so take the latest value per instance before summing across the cluster:

index=kubernetes sourcetype=kubernetes_prometheus source=collectord:* metric_name=pipe
| stats latest(v) as active by host, source, metric_label_name
| stats sum(active) as active by metric_label_name

(On OpenShift: index=openshift sourcetype=openshift_prometheus.) On the demo cluster that returns replace 18, sampling 2, extract 3, whitelist 2, override 6 - annotate a workload and watch the count move; a transform rule that didn’t take effect never shows up. Output selection (devnull) and plain index routing (logs-index) configure destinations rather than pipes and won’t appear here - collectord describe is the check for those. The same source carries file_input_read_bytes (bytes read from container log files) and splunk_post_bytes_sum (bytes actually posted to HEC) - the widening gap between them is your edge filtering at work, measured at the source rather than inferred in Splunk.

What not to cut

The playbook has a floor. Audit trails, security events, authentication logs, and anything with regulatory retention are not volume problems - they are the reason the logging pipeline exists. Give them their own index and retention (lever 6), never sampling (lever 4), and think hard before even filtering (lever 2): dropping “noise” from an audit stream is a decision a compliance officer should co-sign. If cost pressure and compliance collide, the answer is routing regulated data to controlled-retention indexes while cutting the genuinely disposable volume around it - both halves of this playbook, applied to different streams.

Wrap-up

Splunk cost pressure from Kubernetes is not a negotiation with every app team - it is seven annotations’ worth of policy at the collection edge: measure, mute, drop, trim, sample, cap, route. On the demo namespace that was an 82% byte reduction with no loss of searchable signal, and every lever is reversible the moment your needs change.

The full annotation syntax is in the docs: Kubernetes annotations (reference) and OpenShift annotations (reference). Related deep-dives: where annotations should live in a multi-team cluster, and masking PII at the edge - the compliance twin of this post.

If Splunk billing is the reason you are evaluating collection agents, request a trial license and run the measurement searches from this post against your own cluster - the target list usually pays for the afternoon.