Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Linux

Splunk HTTP Event Collector

Configure HTTP Event Collector secure connection

Splunk by default uses self-signed certificates. Collectord provides various configuration options for you to set up how it should connect to HTTP Event Collectord.

Configure trusted SSL connection to the self-signed certificate

If you are using Splunk self-signed certificate, you can copy server CA certificate from $SPLUNK_HOME/etc/auth/cacert.pem and embed it in the image.

Let’s create a basic configuration for the collectord, that accepts the License and specifies the connection to Splunk Enterprise HTTP Event Collector with the self-signed certificate. In this configuration, we define the path to the CA server certificate that collectord should trust and identify the name of the server, specified in the certificate, which is SplunkServerDefaultCert in case of default self-signed certificate.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
caPath = /opt/collectorforlinux/etc/cacert.pem
caName = SplunkServerDefaultCert

With this configuration and cacert.pem, we can place the cacert.pem under /opt/collectorforlinux/etc/cacert.pem and restart collectorforlinux service.

HTTP Event Collector incorrect index behavior

HTTP Event Collector rejects payloads with the indexes that specified Token does not allow to write.

Collectord provides configuration how these errors should be handled with configuration incorrectIndexBehavior.

  • RedirectToDefault - this is the default behavior, which forwards events with an incorrect index to default index of the HTTP Event Collector.
  • Drop - this configuration drops events with incorrect index.
  • Retry - this configuration keeps retrying. Some pipelines, like process stats, can be blocked for the whole host with this configuration.

You can specify behavior with the configuration.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
incorrectIndexBehavior = Drop

Using proxy for HTTP Event Collector

If you need to use a Proxy for HTTP Event Collector, you can define that with the configuration. If you are using SSL connection, you need to include the certificate used by the Proxy

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
proxyUrl = http://proxy.example:4321
caPath = /opt/collectorforlinux/etc/proxie-ca.pem

Using multiple HTTP Event Collector endpoints for Load Balancing and Fail-over

The Collectord can accept multiple HTTP Event Collector URLs for Load Balancing (in case if you are using multiple hosts with the same configuration) and for fail-over.

The collectord provides you with 3 different algorithms for URL selection:

  • random - choose random URL on first selection and after each failure (connection or HTTP status code >= 500)
  • round-robin - choose URL starting from the first one and bump on each failure (connection or HTTP status code >= 500)
  • random-with-round-robin - choose random url on first selection and after that in round-robin on each failure (connection or HTTP status code >= 500)`

The default value is random-with-round-robin

[general]
acceptLicense = true

[output.splunk]
urls.0 = https://hec1.example.com:8088/services/collector/event/1.0
urls.1 = https://hec2.example.com:8088/services/collector/event/1.0
urls.2 = https://hec3.example.com:8088/services/collector/event/1.0

urlSelection = random-with-round-robin

token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

Enable indexer acknowledgement

HTTP Event Collector provides an Indexer acknowledgment, which allows knowing when payload not only accepted by HTTP Event Collector but also written to the Indexer. Enabling this feature can significantly reduce the performance of the clients, including the collectord. But if you need guarantees for data delivery, you can enable it for HTTP Event Collector token and in the collectord configuration.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
ackUrl = https://hec.example.com:8088/services/collector/ack
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
ackEnabled = true
ackTimeout = 3m

Client certificates for collector

If you secure your HTTP Event Collector endpoint with the requirement of client certificates, you can embed them in the image and provide configuration to use them

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
clientCertPath = /conopt/collectorforlinux/etcfig/client-cert.pem
clientKeyPath = /opt/collectorforlinux/etc/client-cert.key

Support for multiple Splunk clusters

If you need to forward logs from the same Docker instance to multiple Splunk Clusters you can configure additional Splunk output in the configuration

[output.splunk::prod1]
url = https://prod1.hec.example.com:8088/services/collector/event/1.0
token = AF420832-F61B-480F-86B3-CCB5D37F7D0D

All other configurations will be used from the default output output.splunk.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.