Installation

Set up Monitoring Docker end to end in under 10 minutes: the Splunk app and HTTP Event Collector first, then Collectord on your hosts, with Docker, Docker Swarm, Amazon ECS or Podman. Collectord forwards container logs, host logs and metrics, enriched with Docker metadata.

If you don’t have a license key yet, request one first: see Before you begin.

Before you begin

• Splunk. Splunk Enterprise 9.2 or later, or Splunk Cloud Platform, for the Monitoring Docker app.
• Docker. Docker Engine 25.0 or later; Docker Engine 20.10 to 24 work too, since Collectord negotiates the API version the daemon speaks. Linux hosts on amd64 or arm64, which Collectord’s images are built for.
• A license key. Request a 30-day evaluation key to try Collectord before production, or a free developer key for a homelab, learning or a personal project (180 days, renewable). Both forms email the key within a few minutes. For a longer evaluation, an offline key or more capacity, contact us.
• Access to registry.outcold.solutions, which the commands below pull from:
  • Image: registry.outcold.solutions/outcoldsolutions/collectorfordocker
  • Pulls go through ghcr.io and pkg-containers.githubusercontent.com, so allow both where egress is filtered.
  • Docker Hub has the same image, with the same tags and digests: outcoldsolutions/collectorfordocker.

Set up Splunk

Install the Monitoring Docker app

Install the latest version of Monitoring Docker from Splunkbase, on your search heads only.

Choose where the stats go

Collectord always sends container and host logs, and Docker events and objects, to an events index. The stats can go to either kind of index: the host, container, process, network, filesystem and disk stats.

Not sure? Start with the events index. You can move later without a gap: set defaultIndexType to both, and the stats go to both indexes while you compare the two sets of dashboards.

Create the indexes

Create the indexes you chose in Settings › Indexes › New Index:

• An events index, with Index Data Type: Events, for example docker. Optional: without one, the data goes to the HEC token’s default index.
• A metrics index, if the stats go there, with Index Data Type: Metrics, for example docker_metrics. The default timestamp resolution works.

In a distributed deployment, create them where your data is indexed: on the indexers, through the cluster manager for an indexer cluster.

Choose the dashboards

The first time you open the app, Splunk shows its App configuration page in front of every dashboard. It stays there until an administrator (the admin or sc_admin role) chooses the dashboard set that matches where the stats go:

1. Open the app, and on the App configuration page select Continue to app setup page. Later, the same page is Setup › Configuration in the app’s menu.
2. Under Dashboards data source, choose Events-index dashboards or Metrics-index dashboards, and select Apply. Do this even when the choice already shows the set you want: until you apply it, the dashboards stay behind the App configuration page.

Both sets stay installed, and the choice only switches the app menu, so you can change it later. On a search head cluster, choose once per member. Installing the app needs no Splunk restart.

The same page holds two more settings: Alert Catalog: who sees the alerts, the roles the alerts you create from the Alert Catalog are shared with (none until you choose), and the Legacy alerts switch for the alerts earlier versions shipped.

Upgrading from an earlier version? Choose the events-index dashboards, and before you select Apply, check the Legacy alerts section: Upgrade has the steps.

Point the app at your indexes

The app finds your data through two macros:

• macro_docker_base reads the events index. Its default, (), names no index, so it searches the indexes your role searches by default. An index you just created isn’t one of them until you add it to the role’s Indexes searched by default (Settings › Roles), so name yours.
• macro_docker_metrics_base reads the metrics index, for the metrics-index dashboards and alerts. It is (index=*) by default, every metrics index your role can search, so it works with any name. Narrow it to yours to keep other metrics out of the app’s searches.

With the indexes from the table above, the two macros read:

1macro_docker_base = (index=docker)
2macro_docker_metrics_base = (index=docker_metrics)

To edit them, open the Macros panel on Setup › Configuration. It links each macro to its editor in Settings › Advanced search › Search macros, where editing needs the admin, power or sc_admin role. Your changes are kept across upgrades.

To split logs and stats across several events indexes, see Splunk indexes.

Create an HTTP Event Collector token

Collectord forwards data to Splunk over the HTTP Event Collector (HEC). If HEC isn’t enabled yet, follow Splunk’s guide to Set up and use HTTP Event Collector in Splunk Web.

Create a token for Collectord in Settings › Data inputs › HTTP Event Collector › New Token:

• Source type: Automatic, the wizard’s default. Collectord assigns its own source types per datatype, and the Splunk app expects them.
• Default index: your events index, if you created one.
• Allowed Indexes: the events index, plus the metrics index if the stats go there. The token refuses any other index, and Collectord then sends that data to the default index instead: the stats would land in the events index without their measurements.

If your indexes aren’t searchable by default, see Splunk Indexes for how to configure them on both the Splunk side and inside Collectord.

The rest of this guide needs two values: the HEC endpoint URL and the token. Check both with curl from a machine on your Docker hosts’ network, a Docker host itself for example: a check from your laptop only proves that your laptop reaches HEC. Splunk answers Success, and the test event stays in the token’s default index:

1$ curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
2{"text": "Success", "code": 0}

-k skips certificate validation; use it only for self-signed certificates.

On Splunk Cloud, HEC has a URL of its own: https://http-inputs-<stack>.splunkcloud.com:443 on a paid stack, https://<stack>.splunkcloud.com:8088 on a trial stack.

Install Collectord for Docker

This section runs Collectord on Docker hosts and Docker Swarm. For Amazon ECS, see Deploy on Amazon ECS, and for Podman, Deploy on Podman.

Check the logging driver

Collectord reads each container’s logs the way its logging driver writes them: from the files of Docker’s json-file driver, or from the systemd journal for the journald driver. Containers that log with any other driver (local, syslog, fluentd and the like) aren’t collected.

A container started with its own driver (docker run --log-driver) is read with that one, and every other container with the daemon’s default. Check the daemon’s default with:

1$ docker info | grep "Logging Driver"
2Logging Driver: json-file

Before 26.10, Collectord read every container’s logs with the daemon’s default driver, so a container started with its own --log-driver had no logs collected.

Collectord doesn’t require you to change the logging driver: it forwards logs from the json-file and journald drivers as Docker writes them. We recommend json-file with rotation (see JSON logging driver configuration). Each container’s log is then a file of its own, and those files are your buffer while Splunk can’t be reached.

To make json-file the daemon’s default, add these keys to /etc/docker/daemon.json (create the file if it’s missing), keeping the ones it already has:

1{
2  "log-driver": "json-file",
3  "log-opts" : {
4    "max-size" : "100m",
5    "max-file" : "3"
6  }
7}

1$ systemctl restart docker

Restarting Docker stops the running containers, unless live-restore is on, and the new options apply only to containers created after it. For the full reference, see Docker’s Configure the default logging driver.

JSON logging driver configuration

Out of the box, Docker doesn’t rotate JSON log files: left alone, they grow until they fill the disk. The max-size and max-file options in the example above set up rotation, so that doesn’t happen.

These files are also your buffer for any disruption between Collectord and Splunk HEC. A container writing 100 MiB per hour with 3 files gives you roughly 3 hours to fix a disruption before the oldest log is overwritten. See Docker’s JSON File logging driver for the full set of options.

Run Collectord

Pull the version of Collectord you want to run:

1docker pull registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1

Pin to a specific version to make upgrades predictable. Follow the blog, Twitter, or the newsletter to keep up with releases.

Run Collectord with the command below. It uses the HEC URL and token from the curl test above. Edit it to set:

• Splunk HEC. The URL, the token, and certificate options if you need them.
• Your license key (see Before you begin), pasted in once you review and accept the license agreement.
• A cluster name, optionally: replace the - on the cluster line. It lets you filter by cluster in the app when you monitor more than one host.
• The metrics index, if you chose it in Choose where the stats go.

If you use a Splunk-generated certificate, Collectord needs some SSL configuration. The simplest path to a working install skips validation with --env "COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true", as the examples below do.

On a host that has been running for a while, Collectord starts by forwarding every historical log still on disk, which can spike both your network and your Splunk indexing. Use the [general] settings thruputPerSecond to cap throughput and tooOldEvents to skip container logs and journal entries older than a given age, for example --env "COLLECTOR__TOOOLDEVENTS=general__tooOldEvents=24h". The lines of the /var/log/*.log files carry no time Collectord reads, so they are dated when it reads them and tooOldEvents doesn’t skip them. On a Splunk trial license (500 MB a day), the first minutes can index more than a day’s allowance.

 1docker run -d \
 2    --name collectorfordocker \
 3    --volume /:/rootfs/:ro \
 4    --volume collector_data:/data/ \
 5    --cpus=1 \
 6    --cpu-shares=204 \
 7    --memory=256M \
 8    --restart=always \
 9    --env "COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0" \
10    --env "COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0"  \
11    --env "COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true"  \
12    --env "COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true" \
13    --env "COLLECTOR__LICENSE=general__license=..." \
14    --env "COLLECTOR__CLUSTER=general__fields.docker_cluster=-" \
15    --privileged \
16    registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1

Running this on Windows? Replace the trailing \ on each line (except the last) with your shell’s own continuation character: a backtick (`) in PowerShell, or ^ in cmd.exe. In PowerShell the backtick has to be the last character on the line, since a trailing space breaks the continuation.

If you chose the metrics index, add these lines to the command, before the image line:

1    --env "COLLECTOR__INDEXTYPE=general__defaultIndexType=metrics" \
2    --env "COLLECTOR__METRICSINDEX=output.splunk__defaultMetricsIndex=docker_metrics" \

Set both. Without defaultMetricsIndex, the stats go to the HEC token’s default index, an events index, and every measurement in them is lost. Compose takes the same values in the service’s environment: list:

1      - COLLECTOR__INDEXTYPE=general__defaultIndexType=metrics
2      - COLLECTOR__METRICSINDEX=output.splunk__defaultMetricsIndex=docker_metrics

A Quadlet unit takes them on Environment= lines (see Metrics index).

For the full set of configuration options, and how to bake your own configuration into a custom image on top of the official one, see the Configuration page.

If you use Docker Compose, the same setup looks like this. Save it as docker-compose.yaml in a directory of its own, readable by root only (chmod 600, since it holds your license key and token), and start Collectord from that directory with docker compose up -d:

 1version: "3.4"
 2services:
 3
 4  collectorfordocker:
 5    image: registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1
 6    volumes:
 7      - /:/rootfs/:ro
 8      - collector_data:/data/
 9    environment:
10      - COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0
11      - COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0
12      - COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true
13      - COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true
14      - COLLECTOR__LICENSE=general__license=...
15      - COLLECTOR__CLUSTER=general__fields.docker_cluster=-
16    restart: always
17    deploy:
18      mode: global
19      restart_policy:
20        condition: any
21      resources:
22        limits:
23          cpus: '1'
24          memory: 256M
25        reservations:
26          cpus: '0.1'
27          memory: 64M
28    privileged: true
29
30volumes:
31  collector_data:
32    name: collector_data

The same file serves Docker Swarm, which reads its version line and deploy.mode: Compose warns that it ignores those two, and applies the resource limits under deploy. The volume keeps the name docker run gives it, collector_data, so Collectord finds its state whichever way you start it.

If your Compose is the old docker-compose 1.x, which ignores deploy, use the file format 2.x instead, with the limits on the service:

 1version: "2.2"
 2services:
 3
 4  collectorfordocker:
 5    image: registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1
 6    volumes:
 7      - /:/rootfs/:ro
 8      - collector_data:/data/
 9    environment:
10      - COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0
11      - COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0
12      - COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true
13      - COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true
14      - COLLECTOR__LICENSE=general__license=...
15      - COLLECTOR__CLUSTER=general__fields.docker_cluster=-
16    restart: always
17    cpus: 1
18    cpu_shares: 204
19    mem_limit: 256M
20    mem_reservation: 64M
21    privileged: true
22
23volumes:
24  collector_data:
25    name: collector_data

Check that Collectord runs

Ask Collectord to check its configuration and its connection to Splunk:

1$ docker exec collectorfordocker /collectord verify

With Compose, run docker compose exec collectorfordocker /collectord verify from the Compose file’s directory. Every check should read OK; Troubleshooting explains the ones that don’t. Then open the Monitoring Docker app in Splunk: dashboards should start populating within a minute or two.

By default, Collectord forwards container logs, host logs (the journal and /var/log/*.log), and metrics for the host, containers, and processes.

Control the container

The docker run command above names the container collectorfordocker:

• Status. docker ps --filter name=collectorfordocker
• Log. docker logs -f collectorfordocker
• Check the setup. docker exec collectorfordocker /collectord verify
• Restart. docker restart collectorfordocker
• Change a setting. The settings are the command’s --env values, which a restart keeps. Remove the container (docker stop collectorfordocker && docker rm collectorfordocker) and run the command again with the new values. Collectord keeps its state in the collector_data volume, so it continues where it stopped.
• Diagnostic bundle for support. docker exec collectorfordocker /collectord diag --stream > diag.tar.gz

With Compose, run them from the Compose file’s directory as docker compose ps, docker compose logs -f collectorfordocker, docker compose exec collectorfordocker /collectord verify and docker compose restart collectorfordocker. After you edit the environment lines, docker compose up -d creates the container again with them.

Deploy on Docker Swarm

For Swarm, reuse the docker-compose.yaml above with stack deploy:

1docker stack deploy --compose-file ./docker-compose.yaml collectorfordocker

The stack runs Collectord as a global service: one task on every node, each forwarding its own node’s logs and metrics. Give each Swarm its own name on the COLLECTOR__CLUSTER line: the dashboards and alerts tell Swarms apart only by it, and unnamed ones all read -.

Run Collectord on the managers

Collectord also forwards the Swarm’s services, tasks and nodes, which only a manager can answer. Docker Swarm has what it sends and which host reads. So Collectord has to run on the managers, and two cases need a hand:

• A drained node runs no stack task, Collectord included. When you drain the leader, another manager that still runs Collectord reads in its place.
• Managers you keep drained, so they do only Swarm’s management work, run no stack task at all. Run Collectord on each of them outside the stack, with the docker run command above, or no host sends the Swarm’s records. A node that runs Collectord both ways forwards its logs twice, so remove its docker run container before you set that manager back to active.

Deploy on Amazon ECS

On Amazon ECS, run Collectord as a daemon service on your EC2 container instances, one task on each. The task definition below is the docker run command above in ECS terms:

• the same image and environment;
• the host’s / mounted read-only at /rootfs;
• a collector_data volume at /data;
• the same limits and privileged.

Edit the same lines (highlighted) and save it as collectorfordocker.json:

 1{
 2  "family": "collectorfordocker",
 3  "requiresCompatibilities": ["EC2"],
 4  "networkMode": "bridge",
 5  "cpu": "1024",
 6  "containerDefinitions": [
 7    {
 8      "name": "collectorfordocker",
 9      "image": "registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1",
10      "essential": true,
11      "cpu": 204,
12      "memory": 256,
13      "privileged": true,
14      "environment": [
15        { "name": "COLLECTOR__SPLUNK_URL", "value": "output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0" },
16        { "name": "COLLECTOR__SPLUNK_TOKEN", "value": "output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" },
17        { "name": "COLLECTOR__SPLUNK_INSECURE", "value": "output.splunk__insecure=true" },
18        { "name": "COLLECTOR__ACCEPTLICENSE", "value": "general__acceptLicense=true" },
19        { "name": "COLLECTOR__LICENSE", "value": "general__license=..." },
20        { "name": "COLLECTOR__CLUSTER", "value": "general__fields.docker_cluster=-" }
21      ],
22      "mountPoints": [
23        { "sourceVolume": "rootfs", "containerPath": "/rootfs", "readOnly": true },
24        { "sourceVolume": "collector_data", "containerPath": "/data" }
25      ]
26    }
27  ],
28  "volumes": [
29    { "name": "rootfs", "host": { "sourcePath": "/" } },
30    {
31      "name": "collector_data",
32      "dockerVolumeConfiguration": { "scope": "shared", "autoprovision": true, "driver": "local" }
33    }
34  ]
35}

Register it, and start it as a daemon service in your cluster:

1aws ecs register-task-definition --cli-input-json file://collectorfordocker.json
2aws ecs create-service --cluster <cluster> --service-name collectorfordocker \
3    --task-definition collectorfordocker --scheduling-strategy DAEMON

The rest of the task definition maps to the docker run options:

• The task’s cpu of 1024 units is --cpus=1, the container’s cpu is --cpu-shares=204, and its memory is --memory=256M.
• ECS starts a new task when one stops, as --restart=always restarts the container.
• The volume’s shared scope keeps it after its task stops, as a named volume outlives its container. So the new task carries on from Collectord’s read positions instead of forwarding the host’s logs again.

Name the cluster on the COLLECTOR__CLUSTER line: the ECS cluster’s name, for example.

Deploy on Podman

Collectord runs on Podman the way it runs on Docker. It reads containers, events and images through Podman’s Docker-compatible API, and container logs from journald, which is Podman’s default logging driver. Use the same collectorfordocker image and the same Monitoring Docker app.

Prepare Podman

• The Podman API socket. Podman has no daemon, so its API is served on demand through a systemd socket, which RHEL and its rebuilds leave disabled. Enable it once:

  bash

  Copy

  1$ systemctl enable --now podman.socket
  2$ ls /run/podman/podman.sock
  3/run/podman/podman.sock

• The journald logging driver. It’s Podman’s default wherever the systemd journal is writable. Check what you’re running with:

  bash

  Copy

  1$ podman info --format '{{.Host.LogDriver}}'
  2journald

  The k8s-file driver (also what Podman calls json-file) truncates its log file in place instead of rotating it. That makes reliable log collection impossible, so Collectord doesn’t read it. If containers.conf sets log_driver = "k8s-file", switch it back to journald.

• Rootless users. From 26.10, Collectord monitors the containers users run with rootless Podman too: see Rootless Podman for what a user’s engine needs.

Run Collectord on Podman

The run command is the Docker one with four changes, highlighted:

• Collectord points at the Podman socket.
• Collectord points at Podman’s storage.
• Its open-files limit is set explicitly, since Podman takes a container’s default from containers.conf (default_ulimits) and its own process limits rather than from a daemon.
• The host is mounted rslave, so the sessions of rootless users who log in after Collectord starts reach it (see Rootless Podman).

 1podman run -d \
 2    --name collectorforpodman \
 3    --volume /:/rootfs/:ro,rslave \
 4    --volume collector_data:/data/ \
 5    --cpus=1 \
 6    --cpu-shares=204 \
 7    --memory=256M \
 8    --restart=always \
 9    --ulimit nofile=1048576:1048576 \
10    --env "COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0" \
11    --env "COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0"  \
12    --env "COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true"  \
13    --env "COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true" \
14    --env "COLLECTOR__LICENSE=general__license=..." \
15    --env "COLLECTOR__CLUSTER=general__fields.docker_cluster=-" \
16    --env "COLLECTOR__GENERALPODMAN_URL=general.docker__url=unix:///rootfs/var/run/podman/podman.sock" \
17    --env "COLLECTOR__GENERALPODMAN_STORAGE=general.docker__dockerRootFolder=/rootfs/var/lib/" \
18    --privileged \
19    registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1

Podman has no daemon to bring containers back after a reboot. --restart=always restarts Collectord if it exits, but at boot only when podman-restart.service is enabled (systemctl enable podman-restart.service). For a production host, run Collectord as a systemd service instead, as shown below.

From 26.10.1, Collectord recognizes Podman from the API’s own responses, not from the socket path. So the socket can also be mounted where Docker’s lives, and the podman-docker package’s /run/docker.sock symlink to the Podman socket works too. collectord verify (podman exec collectorforpodman /collectord verify) reports engine = podman and loggingDriver = journald in its Docker section.

Earlier versions take the socket for Podman’s only when its path contains podman, as in the command above. Mounted any other way, they look for Docker’s log files and forward no container logs.

Run Collectord as a systemd service (Quadlet)

On RHEL and other systemd distributions, the recommended way to run a long-lived container is a Quadlet unit. systemd starts it at boot, restarts it on failure, and its logs are in journalctl -u collectorforpodman. Save the unit as /etc/containers/systemd/collectorforpodman.container:

 1[Unit]
 2Description=Collectord for Monitoring Docker
 3Wants=network-online.target
 4After=network-online.target
 5
 6[Container]
 7ContainerName=collectorforpodman
 8Image=registry.outcold.solutions/outcoldsolutions/collectorfordocker:26.10.1
 9Volume=/:/rootfs/:ro,rslave
10Volume=collector_data:/data/
11Ulimit=nofile=1048576:1048576
12PodmanArgs=--privileged --cpus=1 --memory=256M
13Environment=COLLECTOR__SPLUNK_URL=output.splunk__url=https://hec.example.com:8088/services/collector/event/1.0
14Environment=COLLECTOR__SPLUNK_TOKEN=output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0
15Environment=COLLECTOR__SPLUNK_INSECURE=output.splunk__insecure=true
16Environment=COLLECTOR__ACCEPTLICENSE=general__acceptLicense=true
17Environment=COLLECTOR__LICENSE=general__license=...
18Environment=COLLECTOR__CLUSTER=general__fields.docker_cluster=-
19Environment=COLLECTOR__GENERALPODMAN_URL=general.docker__url=unix:///rootfs/var/run/podman/podman.sock
20Environment=COLLECTOR__GENERALPODMAN_STORAGE=general.docker__dockerRootFolder=/rootfs/var/lib/
21
22[Service]
23Restart=always
24
25[Install]
26WantedBy=multi-user.target

Then load and start it:

1$ systemctl daemon-reload
2$ systemctl start collectorforpodman.service

PodmanArgs carries the options Quadlet has no key for. --privileged has none, and Memory= exists only since Podman 5.5: an older Quadlet refuses to generate a unit with a key it doesn’t know.

The [Install] section is what starts the service at boot. Quadlet generates the unit, so systemctl enable isn’t used.

If systemctl start reports the unit as not found, run /usr/libexec/podman/quadlet -dryrun to see why Quadlet rejected the file.

Rootless Podman

A user who runs containers with rootless Podman has an engine of their own: an API socket at /run/user/<uid>/podman/podman.sock, an image store in their home directory, and their own journal.

From 26.10.1, Collectord finds every user’s socket through [general.docker] rootlessURL.podman, which the default configuration sets to unix:///rootfs/run/user/*/podman/podman.sock. Collectord looks for new sockets every 15 seconds. It monitors those containers next to the system’s, with their logs, container metrics and process metrics. Their events carry docker_container_rootless_uid, the uid of the user whose engine runs them.

A user’s socket exists only while the user’s Podman API socket is enabled and the user has a session. For a service account that is never logged in, turn on lingering:

1$ loginctl enable-linger <user>
2$ systemctl --user -M <user>@ enable --now podman.socket

Mount the host with --volume /:/rootfs/:ro,rslave, as the commands above do. A user’s /run/user/<uid> is a mount of its own, created when the session starts. A private mount (/:/rootfs/:ro) shows Collectord only the sessions that were running when it started, and never a user who logs in later. collectord verify lists the rootless engines it found and warns when the host mount is private.

On RHEL 9 and its rebuilds, systemd gives user sessions only the memory and pids controllers. Their containers then report CPU usage and memory, but not CPU limits, shares and throttling, or I/O. To delegate the rest, add a drop-in and restart the users’ sessions:

1[Service]
2Delegate=cpu cpuset io memory pids

A user’s engine speaks for that user, not for the host that Collectord reads as root, so Collectord holds what it reports to what the kernel confirms:

• Logs come from the journal only, and only from entries that user’s processes wrote. A rootless container that logs with another driver (k8s-file, which rootless Podman falls back to on a volatile journal) isn’t collected, and Collectord logs a warning naming it.
• Stats come from a cgroup inside the user’s slice, and process stats cover only processes running there. Network stats and the socket table are read only while the container’s process is in a network namespace of its own, not the host’s.
• Volume logs (collectord.io/volume.<N>-logs-*) aren’t collected for rootless containers: their volumes are host paths the user chose.
• A rootless container that claims another container’s ID can’t take its place.
• Collectord talks to a user’s socket only while that user’s own process answers on it, whichever path leads there.
• One user’s rootless engines can have Collectord follow at most [general.docker] rootlessMaxContainers running containers, and as many exited ones (500 each by default). Collectord inspects only those, and their events at a bounded rate.

A rootless container’s labels are annotations like any container’s. The user who runs it configures its logs and stats with them, as a Kubernetes user does with pod annotations:

• where its data goes (collectord.io/index, collectord.io/host, collectord.io/output and the rest);
• how its lines are joined, rewritten and sampled;
• how often its stats are taken.

The limits Collectord holds a rootless user to are the things they couldn’t do themselves: read what isn’t theirs, or pass their data off as someone else’s. A rootless container’s events always carry docker_container_rootless_uid and the container’s own id, name and image. Its labels can’t drop those fields, or set or extract fields with the same names.

How much work a rootless user can have Collectord do is bounded coarsely, since an unprivileged user can load the host as much with processes of their own. The bounds are rootlessMaxContainers and what their events carry from their labels:

• Labels over 4 KiB aren’t forwarded.
• One input’s user fields over 4 KiB are dropped.
• An annotation value over 1 KiB is ignored.
• A container whose name or image is longer than 1 KiB isn’t followed.

For a rootless container, thruputPerSecond counts each event whole, fields included.

Not covered yet:

• Collectord forwards the Podman events and objects (docker_events, docker_api inputs) of the system engine only.
• Rootless containers on Podman’s default pasta network have no network metrics, since they have no address of their own.

If unprivileged users must not choose the index, host or sourcetype of what they log, switch rootless discovery off. To stop monitoring rootless engines, set the pattern empty: --env "COLLECTOR__ROOTLESS=general.docker__rootlessURL.podman=".

What Collectord sees on Podman

From 26.10.1, every container the system Podman runs is monitored the same way, logs and metrics, whatever runs it:

• containers started with podman run;
• every container of a pod, including the pod’s infra container, which shows up as a container of its own;
• containers run as systemd services by Quadlet or podman generate systemd;
• containers started with either OCI runtime (crun, runc), and on hosts using either cgroup manager (systemd, cgroupfs).

Earlier versions collect every one of these containers’ logs, but CPU, memory and I/O metrics only for containers started with crun under the systemd cgroup manager. They miss those metrics for containers run by Quadlet (or any systemd unit starting Podman with --cgroups=split), for containers started with runc (RHEL 8’s default), and on hosts using the cgroupfs cgroup manager.

If your configuration overrides [general.docker] containersCgroupFilter, take the libpod part of the default when you upgrade to 26.10.1.

Podman sends no oom event, the one Docker sends when the kernel kills a process in a container for exceeding its memory limit. From Podman 6.0, the die event of a container that had a process OOM-killed carries oomKilled. The Container OOM killed template in the Monitoring Docker app’s Alert Catalog reads it.

Podman keeps that mark on every later die of the container, so the template counts a marked die only when the exit code is 137. A kill the container survives is reported only when the container later dies with exit code 137.

On an earlier Podman, and for rootless engines, whose events Collectord doesn’t forward, the Kernel OOM killer ran template reports the kills from the host’s journal ([input.journald]).

Uninstall

Stop and remove Collectord the way you started it:

• docker run: docker stop collectorfordocker && docker rm collectorfordocker.
• Compose: docker compose down, next to your docker-compose.yaml.
• Swarm: docker stack rm collectorfordocker, plus docker stop collectorfordocker && docker rm collectorfordocker on each drained manager that runs it outside the stack.
• Amazon ECS: delete the daemon service, then deregister its task definition.
• Podman: podman stop collectorforpodman && podman rm collectorforpodman. For the Quadlet unit, stop the service (systemctl stop collectorforpodman.service), delete /etc/containers/systemd/collectorforpodman.container, and run systemctl daemon-reload.

Collectord keeps its state in the collector_data volume on each host. A new Collectord on the same host that mounts the same volume resumes from it and doesn’t send again what was already sent. It finds the volume by name: a Compose file without the volume’s name: collector_data line gets a volume prefixed with the project or stack name, so moving between it and docker run starts from scratch and sends again every log still on disk. To start from scratch on purpose, remove the volume on every host (docker volume ls shows its name).

In Splunk, remove what you added for Collectord once you no longer need it: the app (splunk remove app monitoringdocker), the HEC token, and the indexes with their data.

Next steps

• Create alerts. Pick alerts on the app’s Setup › Alert Catalog page and create the ones relevant to your environment: see Alert Catalog.
• Fix what looks off with the troubleshooting checks.
• Set the timezone. If your Docker hosts run in a timezone other than UTC, the Collectord container needs to know about it. The simplest fix is mounting the host’s timezone file with --volume /etc/localtime:/etc/localtime:ro. Alternatively, set timestampLocation on each [input.files::*] input in the configuration.
• Split your indexes. For better indexing performance, split logs and metrics into separate indexes. Splunk Indexes has the recommended layout and how to wire it up on both sides.
• Extract fields from container logs at search time: see Splunk fields extraction for container logs.
• Forward application logs. Collectord can pick up application logs from inside containers automatically: see annotations.
• Control forwarding per container with annotations: multi-line patterns, field extraction, index, source and sourcetype overrides, dropping noisy lines, and hashing sensitive values.