Common Information Model

The Monitoring OpenShift app maps your nodes, the API audit, OAuth logins and CPU vulnerabilities to the Splunk Common Information Model (CIM). Use it when you run Splunk Enterprise Security, or any search built on the CIM, and want your cluster to show up next to your other servers.

Those searches then show who signed in to a node or to the cluster, who changed what, how busy a node is, and which CPU vulnerabilities its kernel reports.

The app maps the data Collectord already forwards, with eventtypes, tags and search-time fields. Nothing changes at collection, so the mapping adds nothing to your Splunk license. The only cost is the data model acceleration you turn on.

What maps where

No data model has pod, project, container or cluster fields, so the mapping covers what the CIM can describe: nodes, the API audit, logins and CPU vulnerabilities.

Set up the CIM

1. Install the Splunk Common Information Model add-on (Splunk_SA_CIM) on your search heads. Splunk Enterprise Security ships with it.
2. Point the add-on’s index macros for the models you use at the indexes Collectord sends to: cim_Authentication_indexes, cim_Change_indexes, cim_Performance_indexes, cim_Endpoint_indexes, cim_Compute_Inventory_indexes and cim_Vulnerabilities_indexes. List the same indexes as the app’s macro_openshift_base macro, for example (index=openshift): the HEC token’s default index and any index your configuration or annotations name.
3. Keep node stats in an events index. Data models search events, so stats sent only to a metrics index (indexType = metrics) leave Performance empty. The default (events) and both work.

To check the mapping, count a node’s CPU records through the data model:

1| tstats count from datamodel=Performance where nodename=All_Performance.CPU by All_Performance.dest

A data model summary keeps the mapping it was built with. If you accelerated a data model before upgrading the app, rebuild its summary so the records already summarized follow the new mapping: Settings › Data models, then Rebuild on the model.

Nodes

• Performance. dest is the node. Memory, swap and storage are in MB, and uptime is in seconds. cpu_load_percent is the share of the node’s CPU in use (100 minus idle), while the record’s own cpu_user_percent stays in percent of one core. On the stats record, action says whether the node’s clock is synchronized. Filesystems are the container runtime’s alone, unless you set mounts = auto in [input.mount_stats].
• Compute_Inventory. Each Node object the add-on watches (it watches Nodes by default) gives the node as dest and its OS image as os. It also gives the node’s CPU capacity as cpu_count and its memory capacity in MB as mem. Enterprise Security’s System Center then lists every node’s operating system.
• Authentication. sshd’s and sudo’s lines come from the journal (the host logs input). The journal records the command name, systemd unit and uid (syslog_uid) of the process that wrote each line, and the mapping reads them:
  • sshd. The lines a process named sshd, sshd-session or sshd-auth wrote as root (syslog_uid 0) in sshd’s systemd unit. sshd writes as root, so a user’s process named sshd is not mapped. A refusal or a closing line also maps with no unit, where the journal could not name the unit of a process that had already exited; a successful login never does.
  • sudo. The lines a process named sudo wrote. sudo writes as the user who ran it, so the mapping trusts the name. user is the target user, and src_user the user who ran sudo. When a sudo line’s working directory or command repeats sudo’s USER= field, user names every target the line could mean.

Some lines don’t map, or count differently than you might expect:

• The mapping tells the journal’s lines from a log file’s by the source Collectord gives the journal, its directory. If you set a source in [input.journald], sudo’s lines are not mapped.
• An sshd line with an = in it is not mapped: a username or a certificate ID whose key=value text Splunk would read as fields.
• Each sshd line is one event, not one attempt. A wrong password logs a Failed line and, when the client gives up, a closing line. A key an account refuses logs only the closing line.

Collectord sends the command name, unit and uid since 26.10.1, so an older Collectord’s sshd and sudo lines are not mapped.

Listening ports

The listening-ports mapping is off by default, though Collectord still collects the socket table. Once mapped, every listening socket adds an event every 30 seconds to the Endpoint summary that Enterprise Security accelerates: about 127,000 a day for a node with 44 listeners. To map them, enable the openshift_listening_port eventtype in Settings › Event types.

API audit and logins

The mapping reads the API servers’ audit logs as macro_openshift_audit_logs names them. Those are the audit files Collectord reads on the control-plane nodes, from these servers:

• kube-apiserver;
• openshift-apiserver;
• oauth-apiserver;
• the OAuth server.

It judges each line by the source type and source Collectord indexed for it. So a pod that prints "sourcetype":"openshift_host_logs" in its own JSON is not mapped. An override of the macro in local/macros.conf scopes the mapping too. See Audit logs.

The audit records map as follows:

• Change. Every write an API server completed, refused or failed. A forbidden write (403) is a failed change.
  • user is the caller. On an account change, it is the account, and the caller is src_user.
  • object is the object, object_category its resource, and object_path its namespace/name. Where the record names no object, object_path is the namespace alone: a create refused before the API server read it, a generateName create, or a create openshift-apiserver served, such as a route. kube-apiserver passed that create on unread, and the copy that names it is left out.
  • command is the verb, and status is success or failure.
  • dest is the cluster (openshift_cluster, or the node where the cluster is unnamed), and dvc the node that logged it.
• Counted once. openshift-apiserver and oauth-apiserver keep a copy of a request kube-apiserver also recorded. The copy is marked is_proxied_copy and left out. A create in the user API is the exception: it is counted from oauth-apiserver’s record, which names the user, identity or group it made. For an impersonated create, that record names the identity it ran as, not the caller.
• Left out of Change. These records don’t map:
  • leases, events, token and access reviews, token requests, status updates, exec, attach, port-forward and proxy requests (they change no object);
  • dry runs: a delete’s options travel in its body, so a delete is known for a dry run only where the audit policy records request bodies;
  • writes that found nothing to change (404, 409);
  • the namespace controller’s cleanup of a deleted project.
• Account_Management. A service account, user, identity or group created, changed or deleted. user is the account (system:serviceaccount:<namespace>:<name> for a service account, the object’s name for the others), and src_user the caller.
• Authentication. A request no API server could authenticate (401), with no user. Also every login the OAuth server allowed or refused, with the name tried as user.
• Source addresses. src lists every address a record names: the connection’s, and any an X-Forwarded-For header claims.

If you do not forward the kube-apiserver audit files, every record is its own. Where you set macro_openshift_audit_proxied_copy to eval is_proxied_copy=0, also set EVAL-is_proxied_copy = 0 in the [openshift_host_logs] stanza of the app’s local/props.conf.

Splunk ES Content Update

ES Content Update’s Kubernetes detections read the raw audit through its kube_audit macro, not the CIM. Our audit events already carry the fields they read, so point the macro at them and leave the proxied copies out:

1`macro_openshift_audit_logs` NOT stage=RequestReceived NOT is_proxied_copy=1

Four of those detections need the audit policy’s Request level for pods, services and events.

CPU vulnerabilities

Collectord checks /sys/devices/system/cpu/vulnerabilities every hour. While the node reports one, Collectord logs an ALARM-ON "cpu-vulnerabilities" line in its own output, and the Vulnerabilities model reads that line:

• each vulnerable entry as signature and signature_id (spectre_v2, mds, …);
• the CVEs the kernel’s documentation and the vendor’s advisory name for the entry as cve. An entry that stands for a family names all of the family’s: retbleed names AMD’s and Intel’s, whichever the node’s CPU is;
• severity medium, category cpu, and the node as dest.

One event names every vulnerability the node reports, so a search by signature and cve together pairs each vulnerability with every CVE of the event.

Only Collectord’s own container counts: a line macro_openshift_logs_collectord finds, from the project collectorforopenshift. The macro finds Collectord’s image (collectorforopenshift by tag, by digest or with neither) or its container’s name. A workload that names its container or image like Collectord’s, or prints those fields in its own JSON, makes no record.

Two settings need the openshift_cpu_vulnerabilities event type edited (Settings › Event types, linked from Setup › Configuration). Until you edit it, no record is mapped and the Vulnerabilities model stays empty:

• Collectord in another project. Set yours in both namespace terms of the event type.
• A custom [general] fieldsPrefix. Make the event type read the namespace field under your prefix.