Scripted inputs

When the data you need is one command away - the age of the last backup, a license file’s expiry, a vendor tool’s status - a scripted input runs your script on an interval and sends what it prints as events. Each script is one [input.script::<name>] section in Collectord’s configuration. It works in every product this section covers (see Collectord): on Linux and macOS Collectord runs the script on the host, and the container images run it chrooted into the host’s root filesystem (root = /rootfs), so it uses the host’s own shell and tools.

Write the script

A script prints its data on standard output. This one reports the newest backup of the payments database and its age:

1#!/bin/sh
2# The newest payments backup and its age, as one JSON event
3latest=$(ls -t /var/backups/payments/*.tar.gz 2>/dev/null | head -n 1)
4if [ -z "$latest" ]; then
5  echo '{"status":"missing"}'
6  exit 1
7fi
8age=$(( $(date +%s) - $(stat -c %Y "$latest") ))
9printf '{"status":"ok","file":"%s","age_seconds":%d}\n' "$latest" "$age"

Collectord runs the executable directly, never through a shell, so it needs its #! line and an execute bit. Because Collectord runs as root, it runs a script only when the file and every directory above it are owned by root and writable by no one else. It checks again before every run, and refuses a script in /tmp or one mounted from a Kubernetes ConfigMap, whose directory is world-writable. Install the script so it passes:

1sudo mkdir -p /opt/scripts
2sudo install -o root -g root -m 0755 backup-age.sh /opt/scripts/backup-age.sh

The script runs with a fixed environment, never Collectord’s own, which holds the output’s token: PATH, LANG=C, LC_ALL=C, HOME, COLLECTORD_VERSION, COLLECTORD_ENVIRONMENT, COLLECTORD_HOST_ROOT, COLLECTORD_SCRIPT_NAME and COLLECTORD_SCRIPT_INTERVAL (seconds), then the env.<n> pairs you set.

Configure the section

1[input.script::backup_age]
2command = /opt/scripts/backup-age.sh
3interval = 15m

Every event carries the time its run started. A run that exits non-zero still sends what it printed. A run stopped at timeout or maxOutputSize sends nothing with format = event or prometheus, and the complete lines it printed with format = lines; whatever the script left running in its process group is killed. Run a script that needs no root access as an unprivileged user, with user = nobody.

Collectord reads these sections when it starts, so adding or changing one takes a restart. No annotation or container label can name a command: scripts are configuration only. collectord verify reports each section’s executable, user and root, and whether Collectord would run it, without running it. Collectord’s own metrics script_runs (by result), script_run_seconds and script_output_bytes count the runs.

Add a script on Kubernetes and OpenShift

The DaemonSet runs a script on each node it runs on, from the node’s own filesystem: put the script on each node at a host path (with your node image or configuration management), and set root: /rootfs. With Helm, each entry under configuration.inputs.scripts becomes a section (quote arg.<n> and env.<n> keys):

1collectord:
2  configuration:
3    inputs:
4      scripts:
5        backup_age:
6          command: /opt/scripts/backup-age.sh
7          root: /rootfs
8          interval: 15m
9          type: kubernetes_backup_age

helm upgrade restarts the pods, since the chart rolls them on a configuration change. With the manifest, add the section to 002-daemonset.conf in the ConfigMap and restart the DaemonSet’s pods. The Elasticsearch, OpenSearch and syslog charts take the same values.

Add a script on Docker

The image has no shell or tools, so the script lives on the host, and Collectord runs it chrooted into the host’s root filesystem at /rootfs; command, user and workingDirectory are the host’s. Save the section in a configuration file of its own:

1[input.script::backup_age]
2command = /opt/scripts/backup-age.sh
3root = /rootfs
4interval = 15m
5type = docker_backup_age

Mount it next to the shipped configuration by adding --volume /etc/collectorfordocker/100-scripts.conf:/config/100-scripts.conf:ro to the docker run command (or bake it into your image, as Configuration shows), and recreate the container. The installation mounts the host read-only, so the script can read the host’s files but not write them.

Add a script on Linux

Add the section to /opt/collectorforlinux/etc/002-user.conf, then restart Collectord:

1[input.script::backup_age]
2command = /opt/scripts/backup-age.sh
3interval = 15m
4type = linux_backup_age

1sudo systemctl restart collectorforlinux

Monitoring Linux ships one scripted input of its own, enabled: [input.script::package_updates] runs /opt/collectorforlinux/scripts/package-updates.sh every hour as nobody and sends one linux_package_updates event per host with the pending and security updates and whether a reboot is required. The script reads the package manager’s cache (apt, and dnf on the RHEL family and Fedora) without touching the network. On a distribution without a nobody user, set user = (empty) in the section of your 002-user.conf to run it as root. Monitoring Linux shows the result on the Inventory page.

Add a script on macOS

Collectord runs the script with the Mac’s own tools, so leave root out: on macOS Collectord refuses it. The Mac’s tools are BSD’s, so in the example script read the file’s age with stat -f %m instead of stat -c %Y. macOS has no root group, so install the script with wheel:

1sudo mkdir -p "/Library/Application Support/Scripts"
2sudo install -o root -g wheel -m 0755 backup-age.sh "/Library/Application Support/Scripts/backup-age.sh"

Add the section to /opt/collectorformacos/etc/002-user.conf, then restart Collectord’s LaunchDaemon:

1[input.script::backup_age]
2command = /Library/Application Support/Scripts/backup-age.sh
3interval = 15m
4type = macos_backup_age

1sudo launchctl kickstart -k system/com.outcoldsolutions.collectord