Masking PII in OpenShift and Kubernetes logs before it reaches Splunk

Somewhere in your cluster, a service is logging an email address, a card number, or a session token right now. Application code changes weekly, log statements are written under deadline, and code review does not reliably catch a %v that formats a whole customer struct. By the time that line is indexed in Splunk, you have a compliance problem: | delete only hides events from search, it does not remove them from the buckets on disk, and the data has already crossed the network and been replicated.

The reliable place to fix this is the node the container runs on, before the data leaves it. Collectord transforms log lines at the source: mask a value, hash it so it still correlates, drop the line, or re-route it to a restricted index - all driven by annotations that app teams put on their own workloads, and enforceable cluster-wide by the platform team.

This post walks through the whole toolbox on OpenShift with a real cluster and a real Splunk instance behind every example. Everything works identically on Kubernetes - the annotations are the same, substitute kubectl for oc and see the Kubernetes annotations docs instead of the OpenShift ones.

Why masking belongs at the source

You can mask data inside Splunk - SEDCMD in props.conf, or ingest actions on newer versions. Both work, and both share the same weaknesses:

• The raw value already left the node. It crossed the network, sat in forwarder queues, and possibly passed through intermediate heavy forwarders before anything scrubbed it. Under GDPR and HIPAA the transmission itself is part of your data flow.
• The rules are owned centrally. A masking change means a Splunk admin editing props.conf per sourcetype, while the person who actually knows what the payments-api logs look like is the developer who owns the deployment.
• Anything the central rule misses is on disk in an immutable bucket.

Masking at the source inverts all three: the sensitive value never leaves the node, the rules live next to the workload as annotations (in the same Git repository, reviewed in the same pull request as the code that logs), and the platform team can still enforce non-negotiable rules on top with a Configuration CRD or global ConfigMap pipes - more on both below.

The demo: a payments API that logs too much

The examples below run on a local OpenShift cluster (CRC) with Collectord 26.04 forwarding to Splunk. The workload is a payments-api deployment that logs the kind of lines every compliance officer dreads - a customer email, a card number, an SSN, a client IP, and a session token:

1INFO payment authorized user=john.doe@example.com card=4111-1111-1111-1111 amount=42.10 session=sess-8f3a9c2e1b4d
2INFO identity verified user=maria.garcia@example.com ssn=078-05-1120 client=203.0.113.42
3DEBUG healthcheck ok iteration=2
4INFO refund issued user=sam.lee@example.com card=5555-5555-5555-4444 amount=13.37 client=198.51.100.7 session=sess-1a2b3c4d5e6f

Without any configuration, Splunk receives these lines verbatim. Every example that follows is an annotation on the deployment’s pod template - no Collectord restart, no ConfigMap edit, no application change.

Full masking with replace pipes

Replace pipes are pairs of annotations grouped by a number: collectord.io/logs-replace.{N}-search is a regex, collectord.io/logs-replace.{N}-val is the replacement. The simplest use is a full mask - every US Social Security Number becomes a fixed placeholder:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: payments-api
 5  namespace: payments
 6spec:
 7  template:
 8    metadata:
 9      annotations:
10        collectord.io/logs-replace.1-search: '\b\d{3}-\d{2}-\d{4}\b'
11        collectord.io/logs-replace.1-val: 'XXX-XX-XXXX'

Apply it with oc apply (or kubectl apply), and the identity line arrives in Splunk as:

1INFO identity verified user=maria.garcia@example.com ssn=XXX-XX-XXXX client=203.0.113.42

Collectord uses Go’s regexp library (RE2 syntax). regex101.com with the Flavor set to golang is the fastest way to test patterns before you apply them.

Partial masking with named capture groups

A full mask destroys information you may legitimately need. Support teams verify cards by their last four digits; network debugging wants at least the subnet of a client IP. Named capture groups keep the part you need and mask the rest: capture with (?P<name>...) in the search pattern, reference with ${name} in the replacement.

Keep the last four digits of a card number:

1collectord.io/logs-replace.2-search: '\b\d{4}-\d{4}-\d{4}-(?P<last4>\d{4})\b'
2collectord.io/logs-replace.2-val: 'XXXX-XXXX-XXXX-${last4}'

Keep the first octet of an IPv4 address:

1collectord.io/logs-replace.3-search: '\b(?P<octet1>\d{1,3})(\.\d{1,3}){3}\b'
2collectord.io/logs-replace.3-val: '${octet1}.x.x.x'

The result in Splunk:

1INFO refund issued user=sam.lee@example.com card=XXXX-XXXX-XXXX-4444 amount=13.37 client=198.x.x.x session=sess-1a2b3c4d5e6f

Note what did not change: amount=13.37 survived, because the IP pattern demands four octets. Anchor your patterns on structure (separators, digit counts, word boundaries) so that decimals, version numbers, and timestamps are not collateral damage.

Two practical notes:

• Replace pipes apply in name-sorted order, so replace.1 runs before replace.2. The sort is lexicographic - replace.10 sorts before replace.2 - so stick to single digits or zero-pad if you chain more than nine rules.
• The card pattern above matches the hyphen-separated format this service logs. In the wild, card numbers show up with spaces, without separators, and as 15-digit Amex PANs - write patterns against your actual log formats, and test against samples of them.

Hashing: masking that preserves correlation

A masked value is gone - if every email becomes MASKED, you can no longer answer “show me everything this user did”, and that question is exactly what an incident investigation or a GDPR data subject access request needs. Hashing pipes replace the matched value with a deterministic hash: the raw value never reaches Splunk, but the same input always produces the same output, so correlation searches still work.

1collectord.io/logs-hashing.1-match: '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
2collectord.io/logs-hashing.1-function: 'fnv-1a-64'
3collectord.io/logs-hashing.2-match: 'sess-[a-z0-9]+'
4collectord.io/logs-hashing.2-function: 'sha256'

With the replace pipes from above still in place, here is the complete before and after, verified in Splunk:

1INFO payment authorized user=DZEHiXFYRUM card=XXXX-XXXX-XXXX-1111 amount=42.10 session=LfU36knRnA5ycPDRzrw4zNHM4K5zXOjQvmI4qMGAtfU
2INFO refund issued user=5qpanMzmLGE card=XXXX-XXXX-XXXX-4444 amount=13.37 client=198.x.x.x session=TJ2o_JXLAliwVTeXIA8VRtQh6GM3C2cG7BSIeihHLXU

user=5qpanMzmLGE is the fnv-1a-64 hash of sam.lee@example.com; the 43-character session values are sha256. The output is the hash digest encoded as URL-safe base64 with the padding stripped - not hex.

Determinism is the point. Count actions per user without knowing who any user is:

index=openshift sourcetype=openshift_logs "refund issued" | rex "user=(?<user_hash>[\w-]+)" | stats count by user_hash

And when an investigation legitimately needs one specific user’s events, compute the hash of the known identifier outside of Splunk and search for it. For fnv-1a-64:

 1import base64, struct
 2
 3def fnv1a64(value):
 4    h = 0xcbf29ce484222325
 5    for b in value.encode():
 6        h ^= b
 7        h = (h * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
 8    return base64.urlsafe_b64encode(struct.pack('>Q', h)).decode().rstrip('=')
 9
10print(fnv1a64('sam.lee@example.com'))  # 5qpanMzmLGE

index=openshift sourcetype=openshift_logs user=5qpanMzmLGE

That search returns every event for that user - and the email address itself is nowhere in Splunk.

One behavior to design around: the hashing pipe replaces the entire regex match, capture groups are not honored. Write the match pattern so it covers only the value, not its label - the email pattern above matches just the address, which is why user= survives. A pattern like user=[^\s]+ would swallow the user= prefix into the hash too.

Choosing a hash function: FNV vs SHA

The -function annotation accepts 17 algorithms - non-cryptographic checksums and hashes (adler-32, five CRC variants, six FNV variants) and cryptographic ones (md5, sha1, sha256, sha384, sha512). The default, when you omit the annotation, is sha256.

The performance difference is real but small - from the benchmarks in the docs, hashing two IP addresses in a line costs about 1711 ns with fnv-1a-64 against 2220 ns with sha256, roughly 30% more. At typical log volumes neither will show up on a CPU graph, so pick by what the hash needs to do:

• fnv-1a-64 produces a short 11-character value and is the right default for correlation - keeping events groupable without caring who can reverse them.
• sha256 is the right choice when a security or compliance requirement says “cryptographic hash”. It is also what FIPS deployments should standardize on: when Collectord runs in FIPS 140 mode (enabled or enforced), the hashing pipe substitutes sha256 for a requested md5 or sha1 when it builds the pipeline and logs a warning - the non-approved algorithm is never invoked.

Whatever you pick, be honest about what hashing is: Collectord’s hashes are unsalted and deterministic by design (salting would break cross-node correlation). Anyone who can guess a candidate value can compute its hash and confirm it - and for low-entropy values, guessing is enumeration. All one billion possible SSNs can be hashed in seconds; email addresses fall to dictionary attacks. Under GDPR, hashed identifiers are pseudonymized personal data (Article 4(5)), not anonymized data - they still need protection, just less of it. Treat hashing as a control that keeps raw values out of Splunk and limits exposure, not as anonymization.

Forward only what you allow: whitelist

For some workloads the safest policy is inverted: instead of enumerating what to hide, enumerate what to forward. An audit-logger sidecar might emit framework noise, debug output, and the actual audit records - and only the audit records have any business reaching Splunk:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: audit-logger
 5  namespace: payments
 6spec:
 7  template:
 8    metadata:
 9      annotations:
10        collectord.io/logs-whitelist: '^AUDIT '

The container logs four lines per cycle; Splunk receives exactly two:

1AUDIT action=role_grant actor=ops-admin target=serviceaccount/payments-deployer
2AUDIT action=secret_read actor=payments-api secret=payments/stripe-api-key

The INFO and DEBUG lines never leave the node. Beyond the compliance benefit, this is also the bluntest license-cost lever in this post - everything not matching the allowlist is volume you stop paying to ingest.

There is no logs-blacklist annotation for container logs - to drop specific lines while keeping the rest, use a replace pipe with an empty value:

1collectord.io/logs-replace.1-search: '^DEBUG .+$'
2collectord.io/logs-replace.1-val: ''

An event whose message ends up empty is never sent to Splunk, so an empty replacement on a whole-line match deletes the line.

Route sensitive lines to a restricted index: override pipes

Some lines are sensitive even after masking - the fact that an SSN verification happened for some user may itself be regulated data. Override pipes re-route individual matching lines to a different index, source, or sourcetype, while the rest of the container’s output stays where it was:

1collectord.io/logs-override.1-match: 'ssn='
2collectord.io/logs-override.1-index: 'openshift-restricted'
3collectord.io/logs-override.1-type: 'openshift_logs_pii'

Verified result: the identity-verification lines - already masked by the replace pipes - land in the openshift-restricted index with sourcetype openshift_logs_pii, and they are gone from the main openshift index. Override pipes move events, they do not copy them:

1index=openshift-restricted
2INFO identity verified user=3CvL6uiWFuk ssn=XXX-XX-XXXX client=203.x.x.x

On the Splunk side, put the restricted index behind a role that only the security team holds. Index-level access control is the one boundary in Splunk that search cannot cross - a user whose roles do not include openshift-restricted will never see these events, whatever they search.

Three details to get right:

• The HEC token must be allowed to write to the target index. If openshift-restricted is not in the token’s allowed-indexes list, Splunk rejects the write - and Collectord’s shipped default, incorrectIndexBehavior = RedirectToDefault in the [output.splunk] section, then re-sends those events to the token’s default index. For a compliance boundary that is exactly the wrong failure mode: the sensitive lines quietly land in the index everyone can search. Add the index to the token (Settings → Data Inputs → HTTP Event Collector) before you apply the annotation, and consider setting incorrectIndexBehavior = Drop so a future misconfiguration fails closed instead of leaking.
• Override pipes match the transformed text. They run after replace and hashing, so a match pattern written against the raw value (ssn=\d{3}-) will never fire - by the time the override pipe sees the line, the SSN reads XXX-XX-XXXX. Match on what survives the masking, like the ssn= key.
• First match wins. When several override pipes match one line, the first in name-sorted order applies and the rest are skipped.

If a matching line should go to an entirely different Splunk deployment rather than a different index, that is a container-level decision, not a per-line one - see multiple Splunk outputs with collectord.io/logs-output.

What runs when: the pipeline order

Every rule in this post occupies a fixed position in Collectord’s log pipeline, and three of the gotchas above are consequences of that order:

The consequences worth memorizing:

• Whitelist sees raw text. It runs before replace and hashing, so its regex can (and sometimes must) match the PII you are about to mask.
• Extracted fields are not hashed. logs-extraction runs between replace and hashing, and hashing only rewrites the message. If you extract a username into an indexed field and hash it in the message, the plaintext ships in the field. Run masking-by-replace before extraction does its work, or don’t extract sensitive values into fields.
• Override pipes see the final text. Match on structure that survives your own masking.
• Global pipes run last. Whatever a pod annotation did or didn’t do, the ConfigMap-level rules still apply on the way out.

Enforcing masking cluster-wide

Everything so far was an annotation an app team puts on its own deployment - which also means an app team could remove it. Compliance rules need a layer the workload owner cannot override. Collectord has two.

A Configuration CRD with force: true applies annotations to every pod matching a metadata regex, and force makes them win over pod-level annotations. One resource, written by the platform team, masks SSNs in every container of every production namespace:

 1apiVersion: "collectord.io/v1"
 2kind: Configuration
 3metadata:
 4  name: mask-ssn-everywhere
 5  annotations:
 6    collectord.io/logs-replace.1-search: '\b\d{3}-\d{2}-\d{4}\b'
 7    collectord.io/logs-replace.1-val: 'XXX-XX-XXXX'
 8spec:
 9  openshift_namespace: ".+-prod$"
10force: true

The spec keys are the meta fields Collectord attaches to events, so they carry the platform prefix: openshift_namespace on OpenShift, kubernetes_namespace on Kubernetes. Get the prefix wrong and the rule never matches anything - silently. After applying a Configuration, verify it with collectord describe (below) rather than trusting the YAML.

How the CRD layer interacts with pod, workload, and namespace annotations - including the precedence subtleties - is its own post: Layering Collectord annotations.

Global pipes in the ConfigMap are the backstop of last resort. They run at the very end of the pipeline for container logs, host logs, application logs, and events - after and regardless of any per-pod rules. Define them in 001-general.conf, the one file every Collectord workload loads: the shipped template shows a commented example inside 002-daemonset.conf, but the addon Deployment - the pod that forwards events and watched objects - loads only 001-general.conf and 004-addon.conf, so rules placed in a daemonset-only file leave the addon’s data unmasked.

1# mask anything that looks like password=... cluster-wide
2[pipe.replace::passwords]
3patternRegex = (password=)([^\s]+)
4replace = $1********
5
6# hash anything that looks like an IPv4 address cluster-wide
7[pipe.hash::ipv4]
8match = (\d{1,3}\.){3}\d{1,3}
9function = fnv-1a-64

Global replace pipes are available since Collectord 5.21, global hash pipes since 25.10.2. Note the key names differ between the two forms: the replace pipe uses patternRegex / replace, the hash pipe uses match / function (matching the annotation keys).

A sensible division of labor: app teams own precise, format-aware rules on their workloads (they know their log formats); the platform team owns a small set of broad global patterns (passwords, bearer tokens, private IP ranges) plus forced CRDs for regulated namespaces.

Verify what Collectord actually does

Masking rules are compliance controls, so treat them like code: verify after every change.

Ask Collectord for the effective configuration. collectord describe prints the merged annotation set for a pod, and since 26.04 tags every value with its origin - [pod], [namespace], or [configuration:<name>] - so you can prove where a rule comes from:

1oc exec -n collectorforopenshift collectorforopenshift-master-gm8b2 -- \
2  /collectord describe \
3    --namespace payments \
4    --pod payments-api-65f9b9fb78-tsscw \
5    --container payments-api | grep -E 'replace|hashing|override'

1logs-replace.1-search [pod] = \b\d{3}-\d{2}-\d{4}\b
2logs-replace.1-val [pod] = XXX-XX-XXXX
3logs-hashing.1-match [pod] = [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
4logs-hashing.1-function [pod] = fnv-1a-64
5logs-override.1-match [pod] = ssn=
6logs-override.1-index [pod] = openshift-restricted

On Kubernetes, the same command runs with kubectl exec -n collectorforkubernetes ....

Watch for rejected rules. An annotation with an invalid regex is skipped, not applied - Collectord logs a WARN ... invalid annotation line and moves on without the rule. Grep the Collectord pod logs after rolling out new patterns, and check the collectord_errors indexed field in Splunk (index=* collectord_errors=*) for per-event pipe failures.

Search for what should no longer exist. The final test is negative: run searches for the raw patterns (a test card number you deliberately log from a canary pod, an ssn value that is not the mask) and alert if they ever return results. Splunk auto-extracts key=value pairs, so a scheduled alert on index=openshift ssn=* NOT ssn="XXX-XX-XXXX" turns a silent masking regression into a page.

What this gets you for GDPR, HIPAA, PCI DSS, and SOC 2 - and what it doesn’t

What source-side masking legitimately gives your compliance program:

• Data minimization at the point of collection (GDPR Article 5(1)(c)): the log pipeline collects the event without the identifier, rather than collecting everything and scrubbing later.
• Pseudonymization as a technical measure (GDPR Articles 4(5), 32): hashing keeps events correlatable for security monitoring while removing direct identifiers from the analytics platform.
• PAN masking aligned with PCI DSS: displaying at most the last four digits is well inside the requirement that card numbers be masked when displayed and unreadable where stored - and a card number that never reached Splunk is one less system in PCI scope discussions.
• Reduced PHI footprint for HIPAA: every system that holds PHI inherits the full weight of the Security Rule; keeping identifiers out of the logging platform keeps its audit burden proportionate.
• Demonstrable, reviewable controls for SOC 2: the masking rules are YAML in Git, changes go through pull requests, collectord describe proves what is applied, and a negative-search alert monitors the control - exactly the shape auditors like a control to have.

And what it does not give you:

• Regexes only catch what you anticipated. A new log statement with a novel format, a stack trace that embeds a request body, a base64-encoded blob - none of it matches yesterday’s patterns. Masking at the source reduces exposure; it does not make careless logging safe. Keep fixing the logging itself.
• Hashing is not anonymization. Unsalted deterministic hashes of enumerable values (SSNs, phone numbers, emails) are reversible by dictionary. Under GDPR the hashed events remain personal data, with everything that implies for retention and access.
• Ingest-time masking does not clean history. Events indexed before the rule existed are still in Splunk, and their real remedy is retention policy on the bucket level, not | delete.
• This is one control, not a program. You still need index-level RBAC, retention limits, TLS on the forwarding path, and someone who reads the negative-search alert.

One adjacent capability worth knowing about: logs are not the only place secrets leak into Splunk. If you forward watched objects, the modifyValues configuration hashes or removes fields from the objects themselves - modifyValues.object.data.* = hash:sha256 on a Secrets watch ships proof that a Secret changed without shipping the Secret.

Wrap-up

On a cluster with Collectord installed, going from “we log PII into Splunk” to “PII is masked at the node, correlations still work, and the residue is locked in a restricted index” is a handful of annotations on one deployment - no application change, no Splunk-side configuration, no forwarder restart.

Full syntax for everything shown here is in the annotations guides and references: OpenShift (reference) and Kubernetes (reference). For where masking rules should live in a multi-team cluster - pod, namespace, or an enforced Configuration CRD - read Layering Collectord annotations.

If you are evaluating Collectord for a compliance-driven deployment, request a trial license - the masking, hashing, and routing shown here work the same from the first pod you annotate.