Installation

This guide walks you through deploying Collectord to forward Kubernetes container logs, host logs, and audit logs to ElasticSearch or OpenSearch. A typical install takes under 10 minutes. If you don’t have a license yet, you can request a 30-day evaluation.

Install Collectord for Kubernetes / OpenShift

Requirements

Kubernetes 1.24 or later, or OpenShift 4.11 or later. The Helm charts refuse an older cluster.
Linux nodes on amd64 or arm64: Collectord’s images are built for both.
Access to registry.outcold.solutions, which the charts and the manifests pull from:
- Charts: oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-elasticsearch and oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-opensearch
- Image: registry.outcold.solutions/outcoldsolutions/collectorforkubernetes
- Pulls go through ghcr.io and pkg-containers.githubusercontent.com: allow both where egress is filtered.
- Docker Hub has the same charts and image, with the same tags and digests: oci://registry-1.docker.io/outcoldsolutions/<chart>, and docker.io/outcoldsolutions/collectorforkubernetes in collectord.image.repository (or a manifest’s image: lines).

Note for clusters running Collectord for Splunk

If you’re already running Collectord for Splunk on the same cluster, the ElasticSearch deployment runs alongside it. A few details worth knowing up front:

Installed from the manifests, both Collectord instances share the collectorforkubernetes namespace, service account and cluster role, and the manifest applied last sets what both may read. Apply the Collectord for Splunk manifest after the ElasticSearch or OpenSearch one, and again after each upgrade of it: the Splunk manifest’s role also lets a SplunkOutput with tokenFromSecret read its token’s Secret, and the ElasticSearch and OpenSearch manifests’ role does not. The Helm charts install each release into the namespace you name and give it a cluster role of its own.
The ElasticSearch instance ships with annotationsSubdomain set to elasticsearch and the OpenSearch instance with opensearch, so each reads only its own elasticsearch.collectord.io/ or opensearch.collectord.io/ annotations and ignores plain collectord.io/ annotations meant for the Splunk instance. This is what keeps them from stepping on each other’s annotations. An annotation under collectord.collectord.io/ is read by every one of them.
Licensing is shared. Use the same license key for both - running multiple Collectord instances on one cluster doesn’t multiply your usage.
The two use different ports. Both DaemonSets run on the host network of the same nodes, where Collectord’s internal HTTP server - the one the liveness and readiness probes call - binds a port. From 26.10.1 the ElasticSearch deployment binds 127.0.0.1:11891 and the OpenSearch one 127.0.0.1:11892, clear of the Splunk deployment’s 127.0.0.1:11888 and of the 11889 the collectord-splunk-kubernetes chart’s README gives a second Splunk release.
Before 26.10, the Helm charts bound both to 11888, so the one installed second logged failed to start http server ... address already in use at every start. With those charts, set collectord.configuration.general.httpServerBinding to 127.0.0.1:11891 in the ElasticSearch release (11892 in the OpenSearch one), and the port of the DaemonSets’ liveness and readiness probes to the same port.

Install Collectord with the Helm chart for your destination - collectord-elasticsearch or collectord-opensearch - or apply the manifest rendered from it. Either way, you point Collectord at one or more ElasticSearch or OpenSearch hosts, set your license key once you review and accept the license agreement - request an evaluation key with this form if you need one - and, optionally, name the cluster.

With Helm

Helm 3.8 or later. Create my_values.yaml:

my_values.yaml yaml

 1collectord:
 2  configuration:
 3    general:
 4      acceptLicense: true
 5      license: '<your_license_key>'
 6      fields:
 7        "orchestrator.cluster.name": 'development'
 8    outputs:
 9      elasticsearch:
10        default:
11          host: 'https://elasticsearch:9200'
12          authorizationBasicUsername: elastic
13          authorizationBasicPassword: elastic
14          insecure: true

Install the chart:

bash

1$ helm install collectorforkubernetes-elasticsearch \
2    --namespace collectorforkubernetes-elasticsearch \
3    --create-namespace \
4    -f my_values.yaml \
5    oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-elasticsearch

For OpenSearch, install oci://registry.outcold.solutions/outcoldsolutions/charts/collectord-opensearch as collectorforkubernetes-opensearch, in a namespace of that name. Each chart’s README covers the rest of its values.

With the manifest

Grab the latest collectorforkubernetes-elasticsearch.yaml, rendered from the chart. The manifest creates the collectorforkubernetes namespace and deploys every workload Collectord needs. Open it and set the same values in its 001-general.conf:

001-general.conf ini

 1[general]
 2
 3acceptLicense = false
 4
 5license =
 6
 7fields.orchestrator.cluster.name = -
 8
 9...
10
11# ElasticSearch output
12[output.elasticsearch]
13
14host =
15
16authorizationBasicUsername =
17authorizationBasicPassword =
18
19insecure = false

A filled-in example:

001-general.conf ini

 1[general]
 2
 3acceptLicense = true
 4
 5license = ...
 6
 7fields.orchestrator.cluster.name = development
 8
 9...
10
11# ElasticSearch output
12[output.elasticsearch]
13
14host = https://elasticsearch:9200
15
16authorizationBasicUsername = elastic
17authorizationBasicPassword = elastic
18
19insecure = true

Apply the manifest with kubectl:

bash

1$ kubectl apply -f ./collectorforkubernetes-elasticsearch.yaml

Check the workloads

bash

1$ kubectl get all --namespace collectorforkubernetes

With Helm, use the release’s namespace, such as collectorforkubernetes-elasticsearch. Give it a few moments to pull the image and start the containers. Once the pods are Running, head to ElasticSearch or OpenSearch and you should see data arriving.

By default, Collectord forwards container logs, host logs (the journal and /var/log/*.log), and audit logs (when enabled on the API server).

Additional configurations

If you’re deploying onto a cluster that’s been running for a while and has a lot of historical logs on disk, Collectord will start by forwarding all of them - which can spike both your network and your ElasticSearch ingest. Use the [general] settings thruputPerSecond to cap throughput and tooOldEvents to skip events older than a given age.
On ElasticSearch, Collectord submits an index lifecycle policy named logs-collectord that rolls the datastream over to a new backing index every 30 days or at 50 GB per primary shard. It deletes nothing: to keep data for a set time, add a delete phase to es-default-index-lifecycle-management-policy.json.
Collectord submits two index templates - one for the logs-collectord-${COLLECTORD_VERSION} datastream and one for logs-collectord-failed-${COLLECTORD_VERSION}. Review es-default-index-template.json and es-failed-index-template.json and tune the shard and mapping settings for your environment.
The logs-collectord-failed-${COLLECTORD_VERSION} index is where Collectord routes events that fail ingestion into the default index - typically because of a mapping conflict (for example, the type of a field changed and ElasticSearch can no longer index it). Watching this index is a quick way to spot mapping problems.

Managed clusters

Available since Collectord version 26.10

GKE Standard. Collectord runs under the built-in system-node-critical and system-cluster-critical priority classes, and GKE Standard admits those only in a namespace that holds a ResourceQuota for them. The manifest creates that quota, and so does the Helm chart while collectord.priorityClass.resourceQuota is true, the default. Without it, and without a priority class of your own (collectord.priorityClass.name), a new install starts no pod, an upgrade from 26.04 takes Collectord off one node and stops there, and helm without --wait still reports success. To spot it, look for refused pods and a DaemonSet short of its nodes in the release’s namespace (collectorforkubernetes-opensearch for OpenSearch):

bash

1$ kubectl -n collectorforkubernetes-elasticsearch get events --field-selector reason=FailedCreate
2$ kubectl -n collectorforkubernetes-elasticsearch get daemonset

The events read insufficient quota to match these scopes, and the DaemonSet’s CURRENT stays below DESIRED. Set collectord.priorityClass.resourceQuota: true again, and the pods start within a minute or two.

Virtual nodes. The DaemonSet and the addon stay off virtual-kubelet nodes, such as AKS virtual nodes: a virtual node can’t mount the host paths Collectord reads, so those nodes aren’t monitored. The Helm chart keeps them off through collectord.excludeNodes, a map of node labels; set a label to null to allow its nodes again, or add labels of your own:

yaml

1collectord:
2  excludeNodes:
3    type: null                                 # allow virtual-kubelet nodes
4    example.com/no-log-collection: "true"      # keep Collectord off these nodes too

In the manifest, the same labels are the nodeAffinity terms of the DaemonSet and the addon Deployment.

ElasticSearch configuration

You can start using ElasticSearch right away - open Observability -> Logs and you’ll see your data.

OpenSearch configuration

For a fresh OpenSearch installation, you need to create an index pattern before logs are visible:

Go to Management -> Stack Management -> Index Patterns -> Create Index Pattern
Use logs-collectord-* as the index pattern name.
Click Next step
Select @timestamp as the time field name.
Click Create index pattern.