Common Information Model

The Monitoring Kubernetes app maps your nodes, your cluster’s API audit and CPU vulnerabilities to the Splunk Common Information Model (CIM). Use it when you run Splunk Enterprise Security, or any search built on the CIM, and want your cluster to show up next to your other servers.

Those searches then show who signed in to a node, who changed what in the cluster, how busy a node is, and which CPU vulnerabilities its kernel reports.

The app maps the data Collectord already forwards, with eventtypes, tags and search-time fields. Nothing changes at collection, so the mapping adds nothing to your Splunk license. The only cost is the data model acceleration you turn on.

What maps where

No data model has pod, namespace, container or cluster fields, so the mapping covers what the CIM can describe: nodes, the API audit and CPU vulnerabilities.

Set up the CIM

1. Install the Splunk Common Information Model add-on (Splunk_SA_CIM) on your search heads. Splunk Enterprise Security ships with it.
2. Point the add-on’s index macros for the models you use at the indexes Collectord sends to: cim_Authentication_indexes, cim_Change_indexes, cim_Performance_indexes, cim_Endpoint_indexes, cim_Compute_Inventory_indexes and cim_Vulnerabilities_indexes. List the same indexes as the app’s macro_kubernetes_base macro, for example (index=kubernetes): the HEC token’s default index and any index your configuration or annotations name.
3. Keep node stats in an events index. Data models search events, so stats sent only to a metrics index (indexType = metrics) leave Performance empty. The default (events) and both work.

To check the mapping, count a node’s CPU records through the data model:

1| tstats count from datamodel=Performance where nodename=All_Performance.CPU by All_Performance.dest

A data model summary keeps the mapping it was built with. If you accelerated a data model before upgrading the app, rebuild its summary so the records already summarized follow the new mapping: Settings › Data models, then Rebuild on the model.

Nodes

• Performance. dest is the node. Memory, swap and storage are in MB, and uptime is in seconds. cpu_load_percent is the share of the node’s CPU in use (100 minus idle), while the record’s own cpu_user_percent stays in percent of one core. On the stats record, action says whether the node’s clock is synchronized. Filesystems are the container runtime’s alone, unless you set mounts = auto in [input.mount_stats].
• Compute_Inventory. Each Node object the add-on watches (it watches Nodes by default) gives the node as dest and its OS image as os. It also gives the node’s CPU capacity as cpu_count and its memory capacity in MB as mem. Enterprise Security’s System Center then lists every node’s operating system.
• Authentication. sshd’s and sudo’s lines come from the journal (the host logs input). The journal records the command name, systemd unit and uid (syslog_uid) of the process that wrote each line, and the mapping reads them:
  • sshd. The lines a process named sshd, sshd-session or sshd-auth wrote as root (syslog_uid 0) in sshd’s systemd unit. sshd writes as root, so a user’s process named sshd is not mapped. A refusal or a closing line also maps with no unit, where the journal could not name the unit of a process that had already exited; a successful login never does.
  • sudo. The lines a process named sudo wrote. sudo writes as the user who ran it, so the mapping trusts the name. user is the target user, and src_user the user who ran sudo. When a sudo line’s working directory or command repeats sudo’s USER= field, user names every target the line could mean.

Some lines don’t map, or count differently than you might expect:

• The mapping tells the journal’s lines from a log file’s by the source Collectord gives the journal, its directory. If you set a source in [input.journald], sudo’s lines are not mapped.
• An sshd line with an = in it is not mapped: a username or a certificate ID whose key=value text Splunk would read as fields.
• Each sshd line is one event, not one attempt. A wrong password logs a Failed line and, when the client gives up, a closing line. A key an account refuses logs only the closing line.

Collectord sends the command name, unit and uid since 26.10.1, so an older Collectord’s sshd and sudo lines are not mapped.

Listening ports

The listening-ports mapping is off by default, though Collectord still collects the socket table. Once mapped, every listening socket adds an event every 30 seconds to the Endpoint summary that Enterprise Security accelerates: about 127,000 a day for a node with 44 listeners. To map them, enable the kubernetes_listening_port eventtype in Settings › Event types.

API audit

The API audit reaches Splunk once you forward the audit logs. The mapping reads only the trusted sources, as macro_kubernetes_audit_logs names them:

• the kube-apiserver container’s output in kube-system;
• a control-plane node’s audit log files.

It judges each line by the namespace, container, source type and source Collectord indexed for it. So a workload that prints audit-shaped JSON, even JSON naming kube-system and kube-apiserver, is not mapped. An override of the macro in local/macros.conf scopes the mapping too.

The audit records map as follows:

• Change. Every write the API server completed, refused or failed. A forbidden write (403) is a failed change.
  • user is the caller. On an account change, it is the account, and the caller is src_user.
  • object is the object, object_category its resource, and object_path its namespace/name. Where the record names no object, object_path is the namespace alone: a create refused before the API server read it, or a generateName create.
  • command is the verb, and status is success or failure.
  • dest is the cluster (kubernetes_cluster, or the node where the cluster is unnamed), and dvc the node that logged it.
• Left out of Change. These records don’t map:
  • leases, events, token and access reviews, token requests, status updates, exec, attach, port-forward and proxy requests (they change no object);
  • dry runs: a delete’s options travel in its body, so a delete is known for a dry run only where the audit policy records request bodies (level Request);
  • writes that found nothing to change (404, 409);
  • the namespace controller’s cleanup of a deleted namespace.
• Account_Management. A service account created, changed or deleted. user is the account, system:serviceaccount:<namespace>:<name>, and src_user the caller. A service account deleted with its namespace’s collection names no account.
• Authentication. A request no API server could authenticate (401), with no user.
• Source addresses. src lists every address a record names: the connection’s, and any an X-Forwarded-For header claims.

Managed control planes (EKS, GKE, AKS) keep the audit in the cloud, so there is nothing on the nodes to map.

Service account tokens

Token requests stay in the index for the Audit pages. An audit policy that logs serviceaccounts/token at RequestResponse also sends the token it issued, in the response. To keep raw tokens out of Splunk, do one of these:

• keep that resource at Request;
• hash tokens before they leave the node, with a [pipe.hash::tokens] rule whose match is eyJ[\w-]+\.[\w-]+\.[\w-]+.

To keep token requests out of Splunk altogether, add "subresource":"token" to the audit-logs host logs input’s blacklist. The Audit: Service Accounts page’s token panels and the “ServiceAccount token minted by a person” template then go dark.

Splunk ES Content Update

ES Content Update’s Kubernetes detections read the raw audit through its kube_audit macro, not the CIM. Our audit events already carry the fields they read, so point the macro at them:

1`macro_kubernetes_audit_logs` NOT stage=RequestReceived

Four of those detections need the audit policy’s Request level for pods, services and events.

CPU vulnerabilities

Collectord checks /sys/devices/system/cpu/vulnerabilities every hour. While the node reports one, Collectord logs an ALARM-ON "cpu-vulnerabilities" line in its own output, and the Vulnerabilities model reads that line:

• each vulnerable entry as signature and signature_id (spectre_v2, mds, …);
• the CVEs the kernel’s documentation and the vendor’s advisory name for the entry as cve. An entry that stands for a family names all of the family’s: retbleed names AMD’s and Intel’s, whichever the node’s CPU is;
• severity medium, category cpu, and the node as dest.

One event names every vulnerability the node reports, so a search by signature and cve together pairs each vulnerability with every CVE of the event.

Only Collectord’s own container counts: a line macro_kubernetes_logs_collectord finds, from the namespace collectorforkubernetes. The macro finds Collectord’s image (collectorforkubernetes by tag, by digest or with neither) or its container’s name. A workload that names its container or image like Collectord’s, or prints those fields in its own JSON, makes no record.

Two settings need the kubernetes_cpu_vulnerabilities event type edited (Settings › Event types, linked from Setup › Configuration). Until you edit it, no record is mapped and the Vulnerabilities model stays empty:

• Collectord in another namespace. Set yours in both namespace terms of the event type.
• A custom [general] fieldsPrefix. Make the event type read the namespace field under your prefix.