Log forwarding

Choose which of a Mac’s logs Collectord forwards to the events index, and add your own. This page covers what each source sends and how to change it.

Collectord forwards:

• the unified log entries its log packs select;
• a summary of every crash report and kernel panic;
• /var/log/install.log;
• its own log;
• any log file you add.

Unified log

A busy Mac writes hundreds of unified log entries a second, and forwarding them all would be tens of gigabytes a day. So Collectord forwards allowlists instead, called log packs. Each pack is an [input.macos_logs::<pack>] section that selects subsystems and processes. macOS’s log daemon filters the entries before Collectord reads them, so a pack costs about what it delivers.

Four packs ship:

The shipped 001-general.conf lists every subsystem and process each pack selects (see Configuration). An entry two packs select is sent once, under the first by pack name.

Change the packs

Turn a pack on or off in 002-user.conf, or name the packs to turn on with the LogPacks managed preference:

1[input.macos_logs::kernel]
2disabled = false

To forward your own application’s entries, add a pack. Its keys select entries by subsystem (and every subsystem below it) or by process. A key takes the default level, which is default, error and fault entries, unless it names a level:

• .error takes errors and faults only;
• .info adds info entries;
• .debug adds debug entries too.

<n> is any label unique within each key family:

1[input.macos_logs::payments]
2disabled = false
3subsystem.1 = com.example.payments
4process.error.1 = payments-agent

A process key takes everything logged inside the process, its libraries’ entries included. A process key that names an absolute path, such as the security pack’s /usr/bin/sudo, selects that executable alone. It also pins that name for every pack, so another program called sudo cannot pass its lines off as sudo’s.

Keys at info or debug cost more. They read a log stream of their own, and the log store doesn’t keep info and debug entries for Collectord to catch up on after a restart.

To drop lines by their text, add blacklist.<n> patterns to a pack, as a file input filters its lines. To keep only the lines that match a pattern, add whitelist.<n> patterns instead.

The security section of 001-general.conf lists commented-out patterns for what libraries log inside sshd and sudo: “send failed” and the like. On the Macs measured, those lines were 6 to 17% of that pack, and none of them was about who logged in or ran what. To drop those lines, copy the patterns without the ; under [input.macos_logs::security] in your 002-user.conf.

Rate cap and counts

A pack has two rate caps:

• Per template. A template is one process, subsystem, level and format string. Inside a pack, a template over rateLimit entries a minute (60) folds: the rest of the minute becomes one event that says how many entries it stands for, in macos_log_repeats. It keeps folding until a minute ends under the cap.
• Per pack. A pack sends at most packRateLimit entries a minute (300), whatever their templates.

What the caps hold back is counted in the metrics index ([input.macos_log_counts]), so the Log Packs page shows what each pack matched, forwarded and folded. To count the entries in your own searches, weigh each event by what it stands for:

1sourcetype=macos_logs | stats sum(eval(coalesce(macos_log_repeats, 1))) as entries by host, macos_log_pack

For a large laptop fleet where even the packs are too much, set countsOnly = true under [input.macos_logs], or the LogCountsOnly managed preference. Collectord then sends no entries at all, only the per-template counts.

Noise sampler

Before you write a pack, the noise sampler shows you what logs the most on your Macs. Every 15 minutes, Collectord reads 30 seconds of the whole log back from the log store: every process, not only the packs’. It sends:

• the count of the 50 noisiest templates, as format strings, to the metrics index;
• one sample message of each (sourcetype macos_log_samples) to the events index.

The Log Packs page ranks them. noiseSampler.samples = false sends the counts alone, and noiseSampler = false turns the sampler off.

Restarts, sleep and launchd

After a restart, a wake from sleep, or a log stream that ended, Collectord reads the gap back from the log store. It reads from the last entry Splunk acknowledged, at most catchUp (1 hour) back. Up to a second of entries can arrive twice.

macOS never streams launchd’s own entries. So a pack that names the launchd process reads them from the log store every five minutes, and they arrive up to five minutes late.

Private values in the unified log

macOS replaces a value a program logs without marking it public with <private>, for every reader of the log, root included. Most strings and objects are such values: on a measured Mac, 44% of the entries carried one. Log packs and sample messages show those values as <private>, and Collectord cannot turn that off.

A configuration profile that enables private data in the unified log lifts it. It applies to everything logged after you install it, and to every reader of the log on that Mac, not only Collectord. Deploying one is your organization’s decision: through your MDM, or by hand (open the .mobileconfig, then install it in System Settings › General › Device Management):

 1<?xml version="1.0" encoding="UTF-8"?>
 2<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 3<!--
 4  An example, for the organization's MDM at the device level: the unified log keeps the values
 5  programs log without marking them public (most %@ and %s) instead of <private>, from when it is
 6  installed until it is removed, for every reader of the log on the Mac, not only the collector.
 7-->
 8<plist version="1.0">
 9<dict>
10	<key>PayloadContent</key>
11	<array>
12		<dict>
13			<key>PayloadDisplayName</key>
14			<string>Monitoring macOS - Unified log private data</string>
15			<key>PayloadIdentifier</key>
16			<string>com.outcoldsolutions.collectorformacos.logging.system</string>
17			<key>PayloadType</key>
18			<string>com.apple.system.logging</string>
19			<key>PayloadUUID</key>
20			<string>CDBED088-CB88-428A-885A-A753CC0336D3</string>
21			<key>PayloadVersion</key>
22			<integer>1</integer>
23			<key>System</key>
24			<dict>
25				<key>Enable-Private-Data</key>
26				<true/>
27			</dict>
28		</dict>
29	</array>
30	<key>PayloadDisplayName</key>
31	<string>Monitoring macOS - Unified log private data</string>
32	<key>PayloadIdentifier</key>
33	<string>com.outcoldsolutions.collectorformacos.logging</string>
34	<key>PayloadOrganization</key>
35	<string>Outcold Solutions</string>
36	<key>PayloadScope</key>
37	<string>System</string>
38	<key>PayloadType</key>
39	<string>Configuration</string>
40	<key>PayloadUUID</key>
41	<string>F90372B6-E356-426E-BF77-113A1788A4F4</string>
42	<key>PayloadVersion</key>
43	<integer>1</integer>
44</dict>
45</plist>

SSH and sudo

The security pack reads OpenSSH’s authentication lines (“Accepted publickey for”, “Failed password for”, “Invalid user”). OpenSSH writes them at the info level, from sshd-session since macOS 15, so the pack reads sshd-session and sshd at info.

sudo logs the commands it runs only with its log_allowed option, which macOS 15 and later turn off in /etc/sudoers (Defaults !log_allowed). On those Macs, the pack gets sudo’s refusals and never a command it ran.

collectord verify says whether a Mac’s sudo logs its commands, and the Updates and Posture page lists the Macs that don’t. To audit them, set log_allowed back with sudo visudo, or in a file under /etc/sudoers.d.

Fields on log entries

Every entry arrives as a macos_logs event whose text is the message, at the entry’s own time, with these fields:

With [general] fieldsSeparator = ., the fields are sent dotted (macos.log.process), and the app reads both spellings.

For example, to find the errors and faults of the virtual machines over the last day:

1sourcetype=macos_logs macos_log_pack=workloads macos_log_process="com.apple.Virtualization.VirtualMachine" (macos_log_level=Error OR macos_log_level=Fault) earliest=-24h

Crash reports and kernel panics

[input.macos_crash_reports] sends a summary of every crash, hang, jetsam and kernel panic report macOS writes, once each, as a macos_crash_report event. It reads the reports in /Library/Logs/DiagnosticReports and, with userDirectories = true (the default), in every user’s ~/Library/Logs/DiagnosticReports.

The summary is the process, the exception, the crashed thread’s first frames, the processes a jetsam killed, or the panic string. Never the whole file. A report from a user’s folder carries its path, which names the user.

Every minute, the input looks for new reports. It skips a report larger than maxFileSize (8 MB), and on the first start, the reports older than startFromRel (24 hours). The Crashes and Panics page and the kernel panic and virtual machine crash templates of the Alert Catalog read them.

The install log

[input.files::install_log] forwards /var/log/install.log as macos_install_log events, with the process that wrote each entry as syslog_component (installd, softwareupdated, appstoreagent, …). On its first start, Collectord reads the whole file and sends only its last day. The Apps page reads the packages it lists as installed: tools, drivers and system data such as XProtect’s, which are no application.

Your own log files

To forward the log files your applications write, add an [input.files::<name>] section to 002-user.conf. The example below forwards the *.log files under /Library/Logs/payments:

 1[input.files::payments]
 2disabled = false
 3
 4# root location of log files
 5path = /Library/Logs/payments/
 6
 7# glob pattern
 8glob = *.log
 9
10# limit search only on one level
11recursive = false
12
13# how often a file read to its end is checked for more; each check wakes a laptop
14pollingInterval = 5s
15
16# how often path is searched for new files
17walkingInterval = 1m
18
19# override type (source type)
20type = macos_payments
21
22# specify Splunk index
23index =
24
25# a line this does not match continues the event before it
26eventPattern = ^[^\s]

Each file input also takes the extraction, timestamp, sampling and throughput keys the shipped [input.files::install_log] section shows. It has three ways to cut a noisy file’s volume, all off by default:

• grouping repeated lines into one event that counts them;
• folding storms of alike lines into one summary a minute;
• a report of the patterns that cost the most, which the Splunk Usage Report’s Log patterns tab reads.

Each pollingInterval check wakes a laptop, so keep it at seconds, not milliseconds.

Other events

Beside the logs, these inputs send events to the same index:

Several of these name people or programs: Privacy lists them with the switch for each.