Log forwarding

Add your applications’ log files to what Collectord forwards from a Linux host, and choose whether it reads the host’s own logs from the journal or from the syslog files.

Configuration

By default, collectorforlinux forwards the host’s logs from journald and from the *.log files under /var/log. Host logs explains why /var/log/syslog and /var/log/messages are not among them.

Your applications may write logs somewhere else: /opt/<app>/logs/, /srv/<app>/ or a mounted data volume. To pick them up, add an [input.files::<name>] section to 002-user.conf.

The example below forwards *.log files from /opt/myapp/logs. Replace myapp with a name that identifies the source (for example, webportal or payments):

 1[input.files::mylogs]
 2
 3disabled = false
 4
 5# root location of log files
 6path = /opt/myapp/logs
 7
 8# glob pattern
 9glob = *.log
10
11# regex matching pattern (use it instead of glob pattern if you need more complicated filtering)
12# match = 
13
14# limit search only on one level
15recursive = false
16
17# files are read using polling schema, when reach the EOF how often to check if files got updated
18pollingInterval = 250ms
19
20# how often o look for the new files under logs path
21walkingInterval = 5s
22
23# include verbose fields in events (file offset)
24verboseFields = false
25
26# override type (source type)
27type = linux_host_logs
28
29# specify Splunk index
30index =
31
32# regexp to specify the beginning of the event line
33eventPattern = 
34
35# regexp field extraction
36extraction = 
37
38# timestamp field (if field extraction is used)
39timestampField =
40
41# format for timestamp
42# the layout defines the format by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006`
43timestampFormat = Jan 2 15:04:05
44
45# Adjust date, if month/day aren't set in format
46timestampSetMonth = false
47timestampSetDay = false
48
49# timestamp location (if not defined by format)
50timestampLocation = Local
51
52# sample output (-1 does not sample, 20 - only 20% of the logs should be forwarded)
53samplingPercent = -1
54
55# sampling key for hash based sampling (should be regexp with the named match pattern `key`)
56samplingKey =
57
58# set output (splunk or devnull, default is [general]defaultOutput)
59output =
60
61# A cap on the logs forwarded a second, such as 128Kb
62thruputPerSecond =
63
64# Configure events that are too old to be forwarded, for example 168h (7 days) - that will drop all events
65# older than 7 days
66tooOldEvents =
67
68# Configure events that are too new to be forwarded, for example 1h - that will drop all events that are 1h in future
69tooNewEvents =

Host logs: the journal or the syslog files

collectorforlinux reads the host’s own logs from the systemd journal ([input.journald]) and from the *.log files under /var/log ([input.files::logs]). The journal’s entries carry more than a syslog line does: see Fields on journal entries.

It does not read /var/log/syslog or /var/log/messages ([input.files::syslog], off by default). Where rsyslog runs beside journald, as on the RHEL family, Amazon Linux 2023 and Ubuntu, rsyslog writes those files from the journal. Reading both would send every line to Splunk twice. For the same reason, [input.files::logs] leaves out the files rsyslog writes per facility on Debian and Ubuntu (auth.log, kern.log, mail.log and the like).

Before 26.10, [input.files::syslog] was on, so Collectord read /var/log/syslog and /var/log/messages beside the journal.

Every host collectorforlinux installs on runs systemd, but journald can be told to keep no journal: Storage=none in journald.conf forwards everything to rsyslog. On such a host, collectord verify reports x input(journald): FAILED, and Collectord logs a journald input: could not locate journald path warning when it starts.

Read the syslog files there instead: /var/log/syslog or /var/log/messages, and the auth.log Debian and Ubuntu keep authentication in. Turn [input.files::syslog] on and [input.journald] off:

1[input.files::syslog]
2disabled = false
3
4[input.journald]
5disabled = true

Then restart the service with sudo systemctl restart collectorforlinux.

Two more cases to know about:

• The host relays other hosts’ syslog. Lines rsyslog receives over the network never reach the local journal. If rsyslog writes them to files of their own, add an [input.files::<name>] section for those files, as in Configuration. If it mixes them into /var/log/syslog or /var/log/messages, turn [input.files::syslog] on as well; the host’s own lines then arrive twice.
• The journal is kept only in memory. Where there is no /var/log/journal directory, journald keeps the journal under /run/log/journal. Collectord reads it there, but whatever it has not sent yet when the host reboots is gone with it. To keep it, make the journal persistent: sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal && sudo journalctl --flush.

Fields on journal entries

Every entry collectorforlinux reads from the host’s journal ([input.journald], on by default) arrives as a linux_host_logs event with source="/var/log/journal/" and these fields:

journald takes the unit from the writer’s control group. So unlike syslog_component, a program cannot claim another’s unit: search by unit to be sure where a line came from. An entry that has no unit, such as a kernel line, carries no systemd_unit.

A process record (linux_proc_stats_v2) names the unit its process runs in with the same field, so one search finds a unit’s logs and its processes.

With [general] fieldsSeparator = ., Collectord sends the syslog_ fields as syslog.comm, syslog.pid and so on, and the app answers to both spellings. systemd_unit has that one spelling either way.

For example, the error lines of one service over the last day:

sourcetype=linux_host_logs systemd_unit="nginx.service" priority<=3 earliest=-24h

The Services dashboard lists the units that failed, restart in a loop or log errors, with each unit’s log a click away. The Alert Catalog has templates for a failed unit and a restart loop.