Splunk HTTP Event Collector - Monitoring macOS

Collectord sends a Mac’s data to Splunk through the HTTP Event Collector (HEC). Use the [output.splunk] settings on this page to verify HEC’s certificate, go through a proxy, spread the load across endpoints, or send to more than one Splunk cluster.

These settings go in /opt/collectorformacos/etc/002-user.conf, under [output.splunk]. See Configuration for how to apply them.

A configuration profile sets only the URL, the token and insecure, through the SplunkURL, SplunkToken and SplunkInsecure managed preferences. The other settings go in the file, with or without a profile.

Configure HTTP Event Collector secure connection

Splunk ships HEC with a self-signed certificate that no client can verify, so Collectord either accepts it unverified or verifies a certificate of your own.

Verify the HEC certificate

Splunk’s default HEC certificate (SplunkServerDefaultCert) names no host in a Subject Alternative Name, so no client can verify it against the host it connects to. Collectord refuses it with x509: certificate relies on legacy Common Name field, use SANs instead, whatever caPath and caName say. With Splunk’s default certificate, set insecure = true.

To verify the connection instead, give HEC a certificate whose Subject Alternative Name names the host in Collectord’s url, and point caPath at the CA that signed it.

The configuration below accepts the license, points at your HEC URL and verifies its certificate against your CA:

ini

1[general]
2acceptLicense = true
3
4[output.splunk]
5url = https://hec.example.com:8088/services/collector/event/1.0
6token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
7caPath = /opt/collectorformacos/etc/hec-ca.pem

Place the CA at /opt/collectorformacos/etc/hec-ca.pem and restart Collectord with sudo launchctl kickstart -k system/com.outcoldsolutions.collectord. Set caName only when that certificate names the host differently from the url. The name must still be one of the certificate’s Subject Alternative Names.

HTTP Event Collector incorrect index behavior

When Collectord forwards an event to an index that the HEC token isn’t allowed to write to, HEC rejects the payload. The incorrectIndexBehavior setting controls how Collectord handles these rejections:

RedirectToDefault, the default, forwards rejected events to the HEC token’s default index. A stats record redirected to an events index loses its measurements, so allow the token to write the metrics index.
Drop drops rejected events.
Retry keeps retrying. Be careful: a single rejected pipeline (process stats, for example) can stall every other pipeline on the Mac.

Set the behavior in the output configuration:

ini

1[general]
2acceptLicense = true
3
4[output.splunk]
5url = https://hec.example.com:8088/services/collector/event/1.0
6token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
7incorrectIndexBehavior = Drop

Using a proxy for HTTP Event Collector

If Collectord has to reach HEC through a proxy, set proxyUrl. For an SSL connection through the proxy, also include the proxy’s CA certificate:

ini

1[general]
2acceptLicense = true
3
4[output.splunk]
5url = https://hec.example.com:8088/services/collector/event/1.0
6token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
7proxyUrl = http://proxy.example:4321
8caPath = /opt/collectorformacos/etc/proxy-ca.pem

Using multiple HTTP Event Collector endpoints for load balancing and failover

When you have several HEC endpoints fronting the same indexer cluster, Collectord can spread traffic across them and fail over automatically.

Three URL-selection algorithms are available:

random, the default, picks a random URL on first selection, and a different random one after each failure (connection error or HTTP status >= 500).
round-robin starts with the first URL and advances on each failure.
random-with-round-robin picks a random URL on first selection, then advances round-robin on each failure.

With round-robin and random-with-round-robin, the Collectord instances on a failed URL all move to the same next one. So restarting the endpoints one by one can gather most instances on one of them. No algorithm moves an instance back to an endpoint that returns: the endpoint restarted last takes instances only as others fail or Collectord restarts.

ini

 1[general]
 2acceptLicense = true
 3
 4[output.splunk]
 5urls.0 = https://hec1.example.com:8088/services/collector/event/1.0
 6urls.1 = https://hec2.example.com:8088/services/collector/event/1.0
 7urls.2 = https://hec3.example.com:8088/services/collector/event/1.0
 8
 9urlSelection = random
10
11token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

A SplunkURL managed preference replaces every urls.<n> the files name with its one URL. Keep a list of endpoints out of a fleet whose profile sets SplunkURL.

Enable indexer acknowledgement

HEC offers Indexer acknowledgment, which confirms not just that HEC accepted a payload, but that the indexer wrote it. Enable it on both the HEC token and the Collectord output. Each thread keeps up to 64 batches per index in flight while their acknowledgements arrive, so acknowledgement doesn’t limit how fast Collectord sends.

ini

1[general]
2acceptLicense = true
3
4[output.splunk]
5url = https://hec.example.com:8088/services/collector/event/1.0
6ackUrl = https://hec.example.com:8088/services/collector/ack
7token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
8ackEnabled = true
9ackTimeout = 3m

Behind a load balancer, acknowledgement needs each Collectord instance’s connections kept on one indexer, by source-IP or connection affinity. Collectord doesn’t keep cookies, so cookie-based sticky sessions don’t work. Without affinity, batches are posted again before their acknowledgements arrive.

Client certificates for collector

If your HEC endpoint requires mutual TLS, place the client certificate and key on the Mac and point Collectord at them:

ini

1[general]
2acceptLicense = true
3
4[output.splunk]
5url = https://hec.example.com:8088/services/collector/event/1.0
6token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
7clientCertPath = /opt/collectorformacos/etc/client-cert.pem
8clientKeyPath = /opt/collectorformacos/etc/client-cert.key

Support for multiple Splunk clusters

To forward data from the same Mac to more than one Splunk cluster, declare additional [output.splunk::<name>] sections. The example below adds a prod1 output:

ini

1[output.splunk::prod1]
2url = https://prod1.hec.example.com:8088/services/collector/event/1.0
3token = AF420832-F61B-480F-86B3-CCB5D37F7D0D

Anything not set on the named output is inherited from the default output.splunk block. Point an input at it with its output key, or a stats input’s metrics at it with metricsOutput.