Installation

Set up Monitoring Linux end to end in under 10 minutes: the Splunk app and HTTP Event Collector first, then Collectord on your Linux host as a systemd service. Collectord forwards host logs (the journal and /var/log/*.log), host metrics and process metrics.

If you don’t have a license key yet, request one first: see Before you begin.

Before you begin

• Splunk. Splunk Enterprise 9.2 or later, or Splunk Cloud Platform, for the Monitoring Linux app.
• Linux. A host with systemd and glibc 2.28 or later, on amd64 or arm64. glibc 2.28 comes with RHEL 8 and its rebuilds, Debian 10, Ubuntu 20.04, Amazon Linux 2023 and their later releases. Collectord does not run on Amazon Linux 2 or RHEL 7. The packages and the tarball carry the same binary.
• A license key. Request a 30-day evaluation key to try Collectord before production, or a free developer key for a homelab, learning or a personal project (180 days, renewable). Both forms email the key within a few minutes. For a longer evaluation, an offline key or more capacity, contact us.

Set up Splunk

Install the Monitoring Linux app

Install Monitoring Linux from Splunkbase, on your search heads only.

Choose where the stats go

Collectord always sends the journal and log files, failed units and the socket table to an events index. The stats can go to either kind of index: the host, process, network, filesystem and disk stats.

Not sure? Start with the events index. You can move later without a gap: set defaultIndexType to both, and the stats go to both indexes while you compare the two sets of dashboards.

Create the indexes

Create the indexes you chose in Settings › Indexes › New Index:

• An events index, with Index Data Type: Events, for example linux. Optional: without one, the data goes to the HEC token’s default index.
• A metrics index, if the stats go there, with Index Data Type: Metrics, for example linux_metrics. The default timestamp resolution works.

In a distributed deployment, create them where your data is indexed: on the indexers, through the cluster manager for an indexer cluster.

Choose the dashboards

The first time you open the app, Splunk shows its App configuration page in front of every dashboard. It stays there until an administrator (the admin or sc_admin role) chooses the dashboard set that matches where the stats go:

1. Open the app, and on the App configuration page select Continue to app setup page. Later, the same page is Setup › Configuration in the app’s menu.
2. Under Dashboards data source, choose Events-index dashboards or Metrics-index dashboards, and select Apply. Do this even when the choice already shows the set you want: until you apply it, the dashboards stay behind the App configuration page.

Both sets stay installed, and the choice only switches the app menu, so you can change it later. On a search head cluster, choose once per member. Installing the app needs no Splunk restart.

The same page holds two more settings: Alert Catalog: who sees the alerts, the roles the alerts you create from the Alert Catalog are shared with (none until you choose), and the Legacy alerts switch for the alerts earlier versions shipped (see Alert Catalog).

Upgrading from an earlier version? Upgrade the app from Splunkbase, or with Install app from file and Upgrade app checked, and choose the events-index dashboards. Before you select Apply, check that the Legacy alerts section starts with “Upgrading”: the alerts that shipped with earlier versions then stay as they are. The page recognizes an upgrade by this app’s alerts having run at least an hour before; if it reads “New install”, applying the setup switches the old alerts off, so switch them on there first. Then upgrade Collectord.

Point the app at your indexes

The app finds your data through two macros:

• macro_linux_base reads the events index. Its default, (), names no index, so it searches the indexes your role searches by default. An index you just created isn’t one of them until you add it to the role’s Indexes searched by default (Settings › Roles), so name yours.
• macro_linux_metrics_base reads the metrics index, for the metrics-index dashboards and alerts. It is (index=*) by default, every metrics index your role can search, so it works with any name. Narrow it to yours to keep other metrics out of the app’s searches.

With the indexes from the table above, the two macros read:

1macro_linux_base = (index=linux)
2macro_linux_metrics_base = (index=linux_metrics)

To edit them, open the Macros panel on Setup › Configuration. It links each macro to its editor in Settings › Advanced search › Search macros, where editing needs the admin, power or sc_admin role. Your changes are kept across upgrades.

To split logs and stats across several events indexes, set index on each input in 002-user.conf (index.host on [input.system_stats] and [input.proc_stats]). Configuration lists every input.

Create an HTTP Event Collector token

Collectord forwards data to Splunk over the HTTP Event Collector (HEC). If HEC isn’t enabled yet, follow Splunk’s Set up and use HTTP Event Collector in Splunk Web.

Create a token for Collectord in Settings › Data inputs › HTTP Event Collector › New Token:

• Source type: Automatic, the wizard’s default. Collectord assigns its own source types per datatype, and the Splunk app expects them.
• Default index: your events index, if you created one.
• Allowed Indexes: the events index, plus the metrics index if the stats go there. The token refuses any other index, and Collectord then sends that data to the default index instead: the stats would land in the events index without their measurements.

The rest of this guide needs two values: the HEC endpoint URL and the token. Check both with curl from one of your Linux hosts: a check from your laptop only proves that your laptop reaches HEC. Splunk answers Success, and the test event stays in the token’s default index:

1$ curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
2{"text": "Success", "code": 0}

-k skips certificate validation; use it only for self-signed certificates.

On Splunk Cloud, HEC has a URL of its own: https://http-inputs-<stack>.splunkcloud.com:443 on a paid stack, https://<stack>.splunkcloud.com:8088 on a trial stack.

Install Collectord

Collectord for Linux installs from our signed apt and yum repositories at packages.outcold.solutions, on amd64 and arm64. They cover Debian and Ubuntu, RHEL and its rebuilds, Fedora, Amazon Linux 2023 and SUSE. For any other distribution, Collectord comes as a tarball.

The repositories are the simpler path. The package manager checks the signature and installs what Collectord needs. It brings each new release with your next upgrade and keeps your configuration, and it puts the service where systemd looks for it.

Install from the repository

Both repositories are signed with the key at https://packages.outcold.solutions/gpg.key:

1Outcold Solutions LLC (Collectord packages) <support@outcoldsolutions.com>
2RSA 4096, fingerprint 43873B20678AADA89243E0A0B0158B02B251A756

To compare the key you download with the one above, print its fingerprint with curl -fsSL https://packages.outcold.solutions/gpg.key | gpg --show-keys.

On Debian and Ubuntu, add the repository and install. A minimal image may need sudo apt-get install curl gpg first:

1sudo install -d /etc/apt/keyrings
2curl -fsSL https://packages.outcold.solutions/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/outcoldsolutions.gpg
3echo "deb [signed-by=/etc/apt/keyrings/outcoldsolutions.gpg] https://packages.outcold.solutions/apt stable main" | sudo tee /etc/apt/sources.list.d/outcoldsolutions.list
4sudo apt-get update && sudo apt-get install collectorforlinux

On RHEL, Rocky Linux, AlmaLinux, Fedora and Amazon Linux 2023, add the repository and install. The first time, dnf shows the key’s fingerprint and asks before it trusts it:

 1sudo tee /etc/yum.repos.d/outcoldsolutions.repo > /dev/null <<'EOF'
 2[outcoldsolutions]
 3name=Outcold Solutions
 4baseurl=https://packages.outcold.solutions/rpm
 5enabled=1
 6gpgcheck=1
 7repo_gpgcheck=1
 8gpgkey=https://packages.outcold.solutions/gpg.key
 9EOF
10sudo dnf install collectorforlinux

On SUSE, import the key into rpm first, since zypper checks the repository against the keys rpm trusts. Then add the same repository and install:

 1sudo rpm --import https://packages.outcold.solutions/gpg.key
 2sudo tee /etc/zypp/repos.d/outcoldsolutions.repo > /dev/null <<'EOF'
 3[outcoldsolutions]
 4name=Outcold Solutions
 5baseurl=https://packages.outcold.solutions/rpm
 6enabled=1
 7gpgcheck=1
 8repo_gpgcheck=1
 9gpgkey=https://packages.outcold.solutions/gpg.key
10EOF
11sudo zypper install collectorforlinux

Every release stays in the repositories. To install a particular one, name it: collectorforlinux=26.10.1 with apt-get and zypper, collectorforlinux-26.10.1 with dnf.

The package installs Collectord in /opt/collectorforlinux, with the collectorforlinux systemd service. It doesn’t start the service, because Collectord has no license or HEC token yet.

Stay on a release line

The repositories have one channel for every release. So a new release line arrives with your next apt-get upgrade, dnf upgrade or zypper update too. To upgrade only when you choose, hold the package:

1sudo apt-mark hold collectorforlinux         # Debian and Ubuntu
2sudo dnf versionlock add collectorforlinux   # RHEL family, Fedora, Amazon Linux; needs python3-dnf-plugin-versionlock on RHEL
3sudo zypper addlock collectorforlinux        # SUSE

To release the hold, use sudo apt-mark unhold collectorforlinux, sudo dnf versionlock delete collectorforlinux or sudo zypper removelock collectorforlinux.

Install on a host without access to the repository

On a machine that can reach packages.outcold.solutions, download the package for the host’s architecture: uname -m on the host prints x86_64 or aarch64. Copy the package over, and install the file:

• Debian and Ubuntu. Download https://packages.outcold.solutions/apt/pool/main/c/collectorforlinux/collectorforlinux_26.10.1_amd64.deb (_arm64.deb on aarch64). Install it with sudo apt-get install ./collectorforlinux_26.10.1_amd64.deb.
• RHEL family, Fedora, Amazon Linux and SUSE. Download https://packages.outcold.solutions/rpm/packages/collectorforlinux-26.10.1-1.x86_64.rpm (.aarch64.rpm on aarch64). Install it with sudo dnf install ./collectorforlinux-26.10.1-1.x86_64.rpm (sudo zypper install ./collectorforlinux-26.10.1-1.x86_64.rpm on SUSE).

The .rpm is signed, so copy the key from https://packages.outcold.solutions/gpg.key along with it. Once sudo rpm --import gpg.key has imported the key, rpm -K on the package checks the signature.

Or install the tarball

The tarball contains builds for both amd64 and aarch64, and bin/collectord runs the one that matches your host. Download collectorforlinux.tar.gz and extract it into /opt/collectorforlinux. Then create your 002-user.conf from the example it ships:

1curl -fsSL -o /tmp/collectorforlinux.tar.gz https://packages.outcold.solutions/downloads/collectorforlinux/26.10.1/collectorforlinux.tar.gz
2sudo mkdir -p /opt/collectorforlinux
3sudo tar -xvf /tmp/collectorforlinux.tar.gz -C /opt/collectorforlinux
4sudo cp /opt/collectorforlinux/etc/002-user.conf.example /opt/collectorforlinux/etc/002-user.conf
5sudo chmod 600 /opt/collectorforlinux/etc/002-user.conf

The tarball never ships 002-user.conf itself, so extracting a newer one leaves your settings in place.

Configure Collectord

Open /opt/collectorforlinux/etc/002-user.conf with your editor:

1sudoedit /opt/collectorforlinux/etc/002-user.conf

This file holds your overrides for the Collectord defaults. The full default configuration lives in /opt/collectorforlinux/etc/001-general.conf: refer to it when you need to know what options exist.

In 002-user.conf, set:

• Splunk HEC. The URL and the token.
• Your license key (see Before you begin to request one), once you review and accept the license agreement.
• A cluster name, optionally. It lets you filter by cluster in the app when you monitor more than one host group.

For example:

1[general]
2acceptLicense = true
3license = ...
4fields.linux_cluster = dev
5
6[output.splunk]
7url = https://hec.example.com:8088/services/collector/event/1.0
8token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
9insecure = true

If you chose the metrics index, uncomment the block 002-user.conf ships for it:

• defaultMetricsIndex = linux_metrics under [output.splunk], with your metrics index’s name;
• defaultIndexType = metrics under [general].

Set both. Without defaultMetricsIndex, the stats go to the HEC token’s default index, an events index, and every measurement in them is lost.

To confirm the configuration is valid, run collectorforlinux directly from the terminal. Stop it with Ctrl+C before you start the service, because only one Collectord can use its state directory at a time:

1sudo /opt/collectorforlinux/bin/collectorforlinux

Start the collectorforlinux service

If you installed a package, enable and start the service:

1sudo systemctl enable --now collectorforlinux

If you installed the tarball, link the systemd unit it ships first:

1sudo systemctl link /opt/collectorforlinux/bin/collectorforlinux.service
2sudo systemctl daemon-reload
3sudo systemctl enable --now collectorforlinux

Tail the logs to confirm Collectord is running:

1sudo journalctl -fu collectorforlinux

Then let Collectord check its own setup: the configuration, the license, the HEC endpoint and token, and the files and journal it reads.

1sudo /opt/collectorforlinux/bin/collectord verify

See troubleshooting for what each check means.

The unit keeps Collectord out of swap (MemorySwapMax=0). On a host short of memory, moving Collectord’s own memory to swap would turn its garbage collector’s pauses and its allocations into disk reads. Collectord would then fall behind on that host’s logs and stats when they matter most.

The setting covers the whole service, so the scripts its scripted inputs run stay in memory too. On AlmaLinux 9, Collectord holds under 30 MiB, and the hourly pending-updates report peaks at about 160 MiB for under a second while dnf reads its metadata. The kernel can still swap other processes to make room.

systemd applies MemorySwapMax=0 under cgroup v2, the default on RHEL 9, Debian 11, Ubuntu 22.04 and later. It ignores the setting under cgroup v1. If you run Collectord under a unit of your own, add the line to its [Service] section.

Control the service

• Status. sudo systemctl status collectorforlinux
• Log. sudo journalctl -fu collectorforlinux
• Check the setup. sudo /opt/collectorforlinux/bin/collectord verify
• Restart. sudo systemctl restart collectorforlinux. A changed log level or license applies without one; every other setting in 002-user.conf takes a restart.
• Stop and start. sudo systemctl stop collectorforlinux, then sudo systemctl start collectorforlinux.
• Keep it from starting at boot. sudo systemctl disable --now collectorforlinux, which stops it too.
• Diagnostic bundle for support. sudo /opt/collectorforlinux/bin/collectord diag --stream > diag.tar.gz. See Troubleshooting.

Upgrade Collectord

If you get stuck, or want a hand planning an upgrade, contact support - we’ll help and guide you through it.

From the repository, a new release arrives with the rest of the host’s updates, or on its own. The package manager replaces Collectord’s files, keeps your 002-user.conf, and restarts the service if it is running:

1sudo apt-get update && sudo apt-get install --only-upgrade collectorforlinux   # Debian and Ubuntu
2sudo dnf upgrade collectorforlinux                                             # RHEL family, Fedora, Amazon Linux
3sudo zypper update collectorforlinux                                           # SUSE

A package you hold stays where it is until you release it.

If a new version changes the 002-user.conf template, dnf and zypper write it beside yours as 002-user.conf.rpmnew. apt-get asks whether to keep yours (the default) or take the new one.

With the tarball, download the new collectorforlinux.tar.gz, extract it over /opt/collectorforlinux the same way you installed it, and restart the service:

1curl -fsSL -o /tmp/collectorforlinux.tar.gz https://packages.outcold.solutions/downloads/collectorforlinux/26.10.1/collectorforlinux.tar.gz
2sudo tar -xvf /tmp/collectorforlinux.tar.gz -C /opt/collectorforlinux
3sudo systemctl daemon-reload
4sudo systemctl restart collectorforlinux

The tarball replaces only its own files: the binaries, 001-general.conf and the systemd unit. It leaves your 002-user.conf alone.

Either way, Collectord keeps its state in /var/collectord, including how far it has read every log file, so the new version continues where the old one stopped. Removing the package keeps that directory too. sudo apt-get purge collectorforlinux deletes it, along with 002-user.conf.

Upgrade from 26.04 to 26.10

26.10 stops reading /var/log/syslog and /var/log/messages: rsyslog writes them from the journal Collectord already reads, so every line arrived twice. Searches of your own on those sources find nothing new after the upgrade. The upgrade replaces 001-general.conf, so where a host needs the files, set it in 002-user.conf before you upgrade, as Host logs shows:

• A host whose journald keeps no journal (Storage=none): turn [input.files::syslog] on and [input.journald] off.
• A host whose rsyslog also writes logs it receives from other hosts into those files: those lines are not in its journal. Turn [input.files::syslog] on to keep them; the host’s own lines then arrive twice, as before 26.10.

Move a tarball install to a package

Add the repository as Install from the repository shows, but don’t install the package yet. Then stop and disable the service, which also removes the unit link the tarball install created. Install the package over /opt/collectorforlinux, and start the service again:

1sudo systemctl disable --now collectorforlinux
2sudo apt-get update && sudo apt-get install -o Dpkg::Options::=--force-confold collectorforlinux   # or: sudo dnf install collectorforlinux, sudo zypper install collectorforlinux
3sudo chmod 600 /opt/collectorforlinux/etc/002-user.conf
4sudo systemctl enable --now collectorforlinux

Your 002-user.conf stays as it is, and the package’s copy lands beside it as 002-user.conf.dpkg-dist or 002-user.conf.rpmnew. In the commands above:

• --force-confold answers the question apt-get would otherwise ask about the existing file: keep it. Without a terminal, a non-interactive apt-get stops at that question.
• chmod 600 gives your file the mode a package install has, since it holds your license and HEC token.

The 002-user.conf.example left from the tarball is no longer used, and you can delete it.

Uninstall

Remove Collectord the way you installed it:

• Debian and Ubuntu. sudo apt-get remove collectorforlinux stops and removes Collectord and keeps your 002-user.conf and the state in /var/collectord. sudo apt-get purge collectorforlinux removes both too.
• RHEL family, Fedora, Amazon Linux and SUSE. sudo dnf remove collectorforlinux (sudo zypper remove collectorforlinux on SUSE) stops and removes Collectord. It keeps /var/collectord, and saves a 002-user.conf you changed as 002-user.conf.rpmsave.
• The tarball. Stop the service with sudo systemctl disable --now collectorforlinux, then delete /opt/collectorforlinux.

A new install on the same host resumes from the state in /var/collectord and doesn’t send again what was already sent. To start from scratch, delete that directory.

In Splunk, remove what you added for Collectord once you no longer need it: the app (splunk remove app monitoringlinux), the HEC token, and the indexes with their data.

Next steps

• Take the tour of the dashboards.
• Pick alerts on the Alert Catalog page and create the ones relevant to your environment.
• If something looks off, work through the troubleshooting checks.
• Configure log forwarding from custom locations beyond /var/log and journald.