Alert Catalog

The Alert Catalog is where Monitoring Docker alerts come from. Instead of one fixed set of alerts that fires the same way for everyone, you pick from 61 templates, scope them to your clusters, adjust them, and create them as saved searches you own.

Open the catalog from the app menu under Setup › Alert Catalog.

How the page works

The page first looks at your deployment with your own searches:

• which data Collectord sends: container and host logs, host, container and process stats, Docker events, Swarm records, and Collectord’s own metrics;
• whether your stats go to the events index or the metrics index;
• which clusters reported in the last 24 hours;
• whether you may create alerts in the app (the schedule_search capability).

A template whose data is missing says what it needs. To hide those templates, and the ones you already created, turn on Only templates this deployment can install.

The templates are grouped on three tabs by who acts on them:

• Splunk admin: Collectord’s license, its delivery into Splunk, and version upkeep.
• Platform: Collectord on the hosts, the hosts and their kernel, the Docker daemon, the data-root filesystem, and the Swarm’s nodes.
• Application teams: containers (exits, OOM kills, restart loops, health checks, memory and CPU), processes, and Swarm services.

To create alerts:

1. Optionally, pick the Clusters the alerts cover, or leave it empty for every cluster. Each alert you create narrows to those clusters and carries them in its name, so you can create the same template once for prod and once for staging. The scope reads the cluster each Collectord is given in [general] fields.docker_cluster: see Multiple clusters.
2. Select the templates, or Select recommended for the ones on the open tab worth having everywhere.
3. Under More, adjust a template’s parameters: a threshold, a container name pattern.
4. Click Create. Each template becomes a saved search in the Monitoring Docker app, which adds to Triggered Alerts and carries no other action. Attach notifications, such as email, a webhook or your paging tool, on Splunk’s alert page, linked from the results. The Hosts page lists the app’s alerts that fired and have not expired.

A host without fields.docker_cluster still gets a cluster name in the alerts, from its cluster= label or a container’s ECS cluster label. Such a cluster cannot be scoped to: typed in, it matches nothing and the alert never fires. Set fields.docker_cluster on each host to scope its alerts.

An alert runs with its creator’s index access, and anyone who can open it can open its results, whatever indexes their own role may search. So each alert starts private to the user who creates it. Then it is shared with the roles an admin chose under Alert Catalog: who sees the alerts on Setup › Configuration: none until an admin chooses.

When you create alerts, you can pick other roles for them under Alerts created now are visible to. An installed alert’s details say who can open it. Editing and sharing are limited:

• Besides its owner, only admin and sc_admin may edit a shared alert.
• Sharing an alert needs write access to the app’s objects, or the admin_all_objects capability.
• Choosing the app’s roles needs admin_all_objects.

Every template that reads stats or Collectord’s own metrics has two bodies, one for the events index and one for the metrics index. The page installs the one that matches the dashboards chosen on the Configuration page. A template over logs, events or Swarm records has one body, which reads the events index either way.

An installed alert carries its template id and version in its description, so the page recognizes it later. When a newer app version changes the template, the page offers an update.

The update keeps the parameter values the alert was created with, and your actions, severity and throttling. If you edited its search or schedule since, the page shows both versions and lets you keep yours.

Templates

The thresholds below are the defaults; each one is a parameter under More.

Splunk admin

• Collectord stopped reporting (recommended): a host that ran Collectord in the last day has sent none of Collectord’s own metrics for longer than macro_docker_collectord_silent_seconds. The macro is 15 minutes by default, linked from the Configuration page’s Macros panel, and it is the threshold the Collectord usage page uses. Silence means the container is gone, stuck or cannot reach Splunk, and every other alert for that host is blind. The alert fires once an hour per host while the silence lasts, for up to a day.
• Collectord HEC requests failing (recommended): Collectord’s posts to the HTTP Event Collector failed in the last 10 minutes, per host, output and status class.
• Collectord retrying HEC requests: more than 0.25 retries per posted batch over the last 10 minutes, about one post in five failing before it went through.
• Collectord delivery lag: more than 10 percent of posted batches waited over a minute to reach Splunk.
• Collectord license expires within 14 days (recommended), Collectord license checks failing (recommended), Collectord version outdated and Collectord license overuse (recommended).
  • The checks template fires when a Collectord instance could not reach the license server for more than a day, or could not verify its answers for more than an hour. It also fires on a license the server rejected, or one that expired over two days ago.
  • Overuse compares the licensed units Collectord reports with what the license allows: hosts, or CPU cores for a license that counts cores.
• Swarm stopped reporting (recommended): see the Swarm templates below.

Platform

• Collectord on the hosts. Collectord could not parse or had to truncate events and Collectord’s extraction pattern is not matching (more than 10 percent of a container’s lines in an hour) read Collectord’s error tags on the events it forwards. Collectord logged warnings or errors reads its own warnings, and Collectord host diagnostics alarm the host checks it runs (reboot required, CPU vulnerability, low entropy).
• Hosts. Host rebooted reports a host that booted in the last 20 minutes. Host stopped sending stats reports a host whose Collectord runs but whose host stats do not reach Splunk. Docker daemon restarted reports a daemon that started or reloaded, from the host logs.
• Host stats. The templates over the host stats report the conditions below. The load, available memory, swapping, clock, connection tracking, TCP and UDP templates read columns Collectord 26.10.1 and later send.
  • CPU above 90 percent, memory above 85 percent, I/O wait above 30 percent, CPU steal above 10 percent, and swap above 50 percent used;
  • load above 2 per core, memory available below 10 percent, and swapping in over 1 MiB per second;
  • the clock not synchronized;
  • the connection tracking table over 90 percent full, TCP connections turned away by a full accept queue, more than 5 percent of TCP segments retransmitted, and UDP datagrams dropped on full socket buffers;
  • network interface errors or drops, leaving out loopback, the containers’ veth pairs and the bridges Docker and Podman create.
• Disks. Docker data root below 20 percent free (recommended) and Docker data root below 10 percent free inodes read the filesystem holding the Docker data root, where images, writable layers and json-file logs live. Host filesystem below 20 percent free and Host filesystem below 10 percent free inodes watch every other filesystem, one row per host and mount. They need mounts = auto in [input.mount_stats], and the page marks them as needing it until such a filesystem reports.
• Host stalls. Host memory stalled over 10 percent of the time and Host I/O stalled over 20 percent of the time report the share of the time every task with work waited for memory or I/O (full pressure stall information). That is time the host did nothing useful, and it shows before the usage looks alarming. They need Collectord 26.10.1 and a kernel with PSI on: 4.20 and later. RHEL 9 and 10, and the distributions built from them (AlmaLinux, Rocky Linux), keep it off by default, so boot those with psi=1.
• The host’s kernel and systemd. These templates read the host logs (the journal and syslog files):
  • the kernel: Kernel OOM killer ran, Filesystem errors in the kernel log, Tasks blocked in the kernel and Hardware errors in the kernel log. The OOM template sees every kill on the host, rootless containers included, and names the containers and host services whose processes were killed;
  • systemd: A systemd unit failed and A systemd unit restarted 3 times (recommended). A regular expression on the unit name narrows them, for example ^(docker|containerd|sshd)\.service$.
• Docker Swarm nodes. Swarm node down, Swarm manager unreachable, Swarm quorum at risk and Swarm stopped reporting (all recommended), and Swarm node drained or paused. They read the Swarm’s records, which Collectord sends from the Swarm leader, so Collectord has to run on the managers: see Docker Swarm. Name each Swarm with fields.docker_cluster, or they all read - and one still reporting hides another gone silent.

Application teams

• Containers. Container exited with a non-zero code, Container OOM killed, Container restart loop (more than 3 restarts in 15 minutes), Container health check failing and Container memory above 85 percent of its limit are recommended. A Swarm task counts under its slot, <service>.<slot>, so the tasks Swarm starts in its place, on any node, share its alert. The group also has:
  • Container CPU throttled in more than 25 percent of periods: the container hit its CPU limit in more than 25 percent of its CFS enforcement periods, averaged over 5 minutes.
  • Expected container missing: a container that reported stats in the last day has sent none for 15 minutes. Set the name pattern to the containers that must always run.
  • Image pull failing: the Docker daemon or Podman logged image pull errors. A tag the registry does not have counts from Docker Engine 23.0, or 28.0 with the containerd image store.
• Container stalls. Container CPU stalled over 25 percent of the time, Container memory stalled over 10 percent of the time and Container I/O stalled over 20 percent of the time read the container’s own pressure stall information. They need Collectord 26.10.1, cgroup v2 and PSI on, as for the host stall templates, and kernel 5.13 for the CPU share.
• Processes. Process using over 80 percent of its open-file limit reports the file descriptors a process holds against its soft limit (ulimit -n, a container’s --ulimit nofile). At the limit, its next open or accept fails. Narrow it with a pattern on the process name. It needs Collectord 26.10.1 or later.
• Docker Swarm services. Swarm service below its desired tasks, Swarm tasks failing (3 or more failed or rejected tasks of a service in 15 minutes) and Swarm service update stuck, all recommended.

On Podman, Container OOM killed reads the kills Podman 6.0 and later mark on a container’s die event. Kernel OOM killer ran covers earlier releases and rootless engines: see What Collectord sees on Podman.

A template can sit on more than one tab:

• The Application teams tab also holds the Collectord parse and extraction templates and Kernel OOM killer ran.
• The Platform tab also holds most of the Application teams tab’s container templates, the Swarm service templates, Collectord stopped reporting, and the HEC, license-check and version templates.

Shipped alerts

Earlier versions of Monitoring Docker shipped 13 predefined alerts. They stay in this release, enabled and marked deprecated in their description, and a later release removes them. Each has a replacement template:

An upgrade keeps them: the shipped alerts keep running with the actions you attached, as long as the Configuration page recognizes the upgrade, its Legacy alerts section reading “Upgrading” (see Upgrade). They read the events index only. Once Collectord sends its stats and metrics to a metrics index alone (indexType = metrics), the ones that watch them find nothing, while their replacement templates read either index.

The Shipped alerts tab on the catalog page lists the ones you customized, beside their replacement templates. The Configuration page’s Legacy alerts section switches them all on or off.

With Keep as my own copies (the admin_all_objects capability), the same section writes each one in full into the app’s local configuration, actions and enabled state included. Each copy survives the release that removes them. On a new install, setup finishes with the legacy alerts off, and you pick alerts on the catalog page.