Monitoring Linux

Common Information Model

Map Linux hosts to the Splunk Common Information Model for Splunk Enterprise Security: sshd and sudo, performance, filesystems, listening ports and pending updates.

Available since Collectord version 26.10

When you run Splunk Enterprise Security, or any search built on the Splunk Common Information Model (CIM), you want your Linux hosts to show up there. The Monitoring Linux app maps the data Collectord already forwards to the CIM data models, with eventtypes, tags and search-time fields.

The mapping shows who signed in over SSH, who ran sudo, how busy a host is, how full its filesystems are, and which package updates are pending. Nothing changes at collection, so the mapping adds nothing to your Splunk license. The only cost is the data model acceleration you turn on.

What maps where

Data modelFromEventtype
Authenticationsshd’s logins and refusals, from the journallinux_ssh_authentication
Authentication (privileged)sudo, from the journallinux_sudo
Performance: CPU, Memory, Uptime, Timesyncevery host stats recordlinux_host_performance
Performance: Storageevery filesystem recordlinux_filesystem
Endpoint: Portsthe sockets a host listens on (off by default)linux_listening_port
Compute_Inventory: CPU, Memory, OSevery host stats recordlinux_host_inventory
Compute_Inventory: Virtual_OSthe hypervisor a host runs under, where it names onelinux_host_virtual
Updateseach package a pending-updates report listslinux_updates_pending

Set up the CIM

  1. Install the Splunk Common Information Model add-on (Splunk_SA_CIM) on your search heads. Splunk Enterprise Security ships with it.
  2. Point the add-on’s index macros for the models you use at the indexes Collectord sends to. The macros are cim_Authentication_indexes, cim_Performance_indexes, cim_Endpoint_indexes, cim_Compute_Inventory_indexes and cim_Updates_indexes. List the same indexes as the app’s macro_linux_base macro: the HEC token’s default index and any index your configuration names, for example (index=linux).
  3. Keep host stats in an events index. Data models search events, so stats sent only to a metrics index (indexType = metrics) leave Performance and Compute_Inventory empty. The default (events) and both work.

To check the mapping, count a host’s CPU records through the data model:

text
1| tstats count from datamodel=Performance where nodename=All_Performance.CPU by All_Performance.dest

Performance

dest is the host. In the Performance data model:

  • Memory, swap and storage are in MB, and uptime in seconds.
  • cpu_load_percent is the share of the host’s CPU in use (100 minus idle). The record’s own cpu_user_percent stays in percent of one core.
  • On the stats record, action says whether the host’s clock is synchronized.
  • Storage takes size, used and free space in MB and percent, df-style, with the mount point as mount.

The same stats record maps to Compute_Inventory, so Enterprise Security’s System Center lists every host’s operating system:

  • the host’s operating system as os: the distribution’s name, such as Ubuntu 24.04.4 LTS, or the kernel type where the host sends none;
  • cpu_count, and mem in MB;
  • the hypervisor as hypervisor, where the host runs under one.

SSH and sudo

sshd’s and sudo’s lines come from the journal (the [input.journald] input). The journal records the command name, systemd unit and uid (syslog_uid) of the process that wrote each line:

  • sshd’s lines are the ones a process named sshd, sshd-session or sshd-auth wrote as root (syslog_uid 0) in sshd’s systemd unit. sshd writes as root, so a user’s process named sshd is not mapped. A refusal or a closing line also maps with no unit, where the journal could not name the unit of a process that had already exited; a successful login never does.
  • sudo’s lines are the ones a process named sudo wrote. sudo writes as the user who ran it, so the mapping trusts the name. They map to Privileged_Authentication, with the user who ran sudo as src_user and the target user as user. A sudo line whose working directory or command repeats sudo’s USER= field names every target it could mean as the user.
  • Collectord sends the command name, unit and uid since 26.10.1, so an older Collectord’s sshd and sudo lines are not mapped.
  • The mapping tells the journal’s lines from a log file’s by the source Collectord gives the journal, its directory. If you set a source in [input.journald], sudo’s lines are not mapped.
  • An sshd line with an = in it (a username or a certificate ID whose key=value text Splunk would read as fields) is not mapped.
  • Each sshd line is one event, not one attempt. A wrong password logs a Failed line and, when the client gives up, a closing line. A key an account refuses logs only the closing line.

Pending updates

The [input.script::package_updates] input reports the host’s pending package updates every hour. Each package a report lists maps to Updates, so Enterprise Security’s Update Center lists them:

  • status: available;
  • signature and signature_id: the package name;
  • vendor_product: the package manager (dnf, apt, …);
  • dest: the host.

An updated package keeps its available row until the last report that listed it ages out of the Update Center’s time range.

Listening ports

The listening-ports mapping is off by default, though Collectord still collects the socket table. Every listening socket adds an event every 30 seconds to the Endpoint summary that Enterprise Security accelerates: about 127,000 a day for a host with 44 listeners. To map them, enable the linux_listening_port eventtype in Settings > Event types.