Monitoring Docker

Common Information Model

Map Docker hosts, container restarts and CPU vulnerabilities to the Splunk Common Information Model for Splunk Enterprise Security.

Available since Collectord version 26.10

The Monitoring Docker app maps your Docker hosts, container restarts and CPU vulnerabilities to the Splunk Common Information Model (CIM). Use it when you run Splunk Enterprise Security, or any search built on the CIM, and want your Docker hosts to show up next to your other servers.

Those searches then show who signed in to a host, how busy it is, which containers were restarted, and which CPU vulnerabilities its kernel reports.

The app maps the data Collectord already forwards, with eventtypes, tags and search-time fields. Nothing changes at collection, so the mapping adds nothing to your Splunk license. The only cost is the data model acceleration you turn on.

What maps where

Data modelFromEventtype
Authenticationsshd’s logins and refusals on a host, from the journaldocker_ssh_authentication
Authentication (privileged)sudo on a host, from the journaldocker_sudo
Change: Endpoint_Restartsa container restarted through the API (docker restart)docker_container_restart
Performance: CPU, Memory, Uptime, Timesyncevery host stats recorddocker_host_performance
Performance: Storageevery filesystem recorddocker_filesystem
Endpoint: Portsthe sockets a host listens on (off by default)docker_listening_port
Compute_Inventory: CPU, Memory, OSevery host stats recorddocker_host_inventory
Compute_Inventory: Virtual_OSthe hypervisor a host runs under, where it names onedocker_host_virtual
Vulnerabilitiesthe CPU vulnerabilities Collectord’s hourly check reports for a hostdocker_cpu_vulnerabilities

Set up the CIM

  1. Install the Splunk Common Information Model add-on (Splunk_SA_CIM) on your search heads. Splunk Enterprise Security ships with it.
  2. Point the add-on’s index macros for the models you use at the indexes Collectord sends to: cim_Authentication_indexes, cim_Change_indexes, cim_Performance_indexes, cim_Endpoint_indexes, cim_Compute_Inventory_indexes and cim_Vulnerabilities_indexes. List the same indexes as the app’s macro_docker_base macro, for example (index=docker): the HEC token’s default index and any index your configuration or container labels name.
  3. Keep host stats in an events index. Data models search events, so stats sent only to a metrics index (indexType = metrics) leave Performance and Compute_Inventory empty. The default (events) and both work.

To check the mapping, count a host’s CPU records through the data model:

text
1| tstats count from datamodel=Performance where nodename=All_Performance.CPU by All_Performance.dest

A data model summary keeps the mapping it was built with. If you accelerated a data model before upgrading the app, rebuild its summary so the records already summarized follow the new mapping: Settings › Data models, then Rebuild on the model.

Hosts

  • Performance. dest is the host. Memory, swap and storage are in MB, and uptime is in seconds. cpu_load_percent is the share of the host’s CPU in use (100 minus idle), while the record’s own cpu_user_percent stays in percent of one core. On the stats record, action says whether the host’s clock is synchronized. Filesystems are Docker’s alone, unless you set mounts = auto in [input.mount_stats].
  • Compute_Inventory. The same stats record gives the host’s operating system as os: the distribution’s name, such as Ubuntu 24.04.4 LTS, or the kernel type where the host sends none. Beside it come cpu_count, mem in MB, and the hypervisor as hypervisor where the host runs under one. Enterprise Security’s System Center then lists every host’s operating system.
  • Authentication. sshd’s and sudo’s lines come from the journal (the host logs input). The journal records the command name, systemd unit and uid (syslog_uid) of the process that wrote each line, and the mapping reads them:
    • sshd. The lines a process named sshd, sshd-session or sshd-auth wrote as root (syslog_uid 0) in sshd’s systemd unit. sshd writes as root, so a user’s process named sshd is not mapped. A refusal or a closing line also maps with no unit, where the journal could not name the unit of a process that had already exited; a successful login never does.
    • sudo. The lines a process named sudo wrote. sudo writes as the user who ran it, so the mapping trusts the name. user is the target user, and src_user the user who ran sudo. When a sudo line’s working directory or command repeats sudo’s USER= field, user names every target the line could mean.

Some lines don’t map, or count differently than you might expect:

  • The mapping tells the journal’s lines from a log file’s by the source Collectord gives the journal, its directory. If you set a source in [input.journald], sudo’s lines are not mapped.
  • An sshd line with an = in it is not mapped: a username or a certificate ID whose key=value text Splunk would read as fields.
  • Each sshd line is one event, not one attempt. A wrong password logs a Failed line and, when the client gives up, a closing line. A key an account refuses logs only the closing line.

Collectord sends the command name, unit and uid since 26.10.1, so an older Collectord’s sshd and sudo lines are not mapped.

Listening ports

The listening-ports mapping is off by default, though Collectord still collects the socket table. Once mapped, every listening socket adds an event every 30 seconds to the Endpoint summary that Enterprise Security accelerates: about 127,000 a day for a host with 44 listeners. To map them, enable the docker_listening_port eventtype in Settings › Event types.

Container restarts

A container restarted through the API (docker restart, or a restart from Docker Compose) maps to Change’s Endpoint_Restarts:

  • action modified and change_type restart;
  • the container’s name as object and its ID as object_id;
  • the host as dest.

A restart policy restarting a container logs a die and a start, not a restart, so those restarts are not mapped.

CPU vulnerabilities

Collectord checks /sys/devices/system/cpu/vulnerabilities every hour. While the host reports one, Collectord logs an ALARM-ON "cpu-vulnerabilities" line in its own output, and the Vulnerabilities model reads that line:

  • each vulnerable entry as signature and signature_id (spectre_v2, mds, …);
  • the CVEs the kernel’s documentation and the vendor’s advisory name for the entry as cve. An entry that stands for a family names all of the family’s: retbleed names AMD’s and Intel’s, whichever the host’s CPU is;
  • severity medium, category cpu, and the host as dest.

One event names every vulnerability the host reports, so a search by signature and cve together pairs each vulnerability with every CVE of the event.

Only Collectord’s own container counts: a line macro_docker_logs_collectord finds, from a container the Docker or Podman daemon runs as root, never a rootless user’s. The macro finds Collectord’s image (collectorfordocker by tag, by digest or with neither) or the container name collectorfordocker.

A container that names itself or its image like Collectord’s, or prints those fields in its own JSON, makes no record unless it runs rootful under that name.

With a custom [general] fieldsPrefix, add NOT <prefix>container_rootless_uid=* to the docker_cpu_vulnerabilities eventtype, which reads only the default field names.