Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

Monitoring Kubernetes

Configuration

collectorforkubernetes.yaml

Configuration files for previous releases of Collectord deployment files can be found at github.com/outcoldsolutions/collectord-configurations

Created Kubernetes Objects

The configuration file collectorforkubernetes.yaml creates several Kubernetes Objects.

  • Namespace collectorforkubernetes.
  • ClusterRole collectorforkubernetes with limited capabilities to get, list, and watch deployed objects. Collectord uses this information to enrich logs and stats with Kubernetes-specific metadata.
  • ServiceAccount collectorforkubernetes is used to connect to the Kubernetes API.
  • ClusterRoleBinding collectorforkubernetes to bind the service account to the cluster role.
  • ConfigMap collectorforkubernetes delivers configuration files for collectord.
  • DaemonSet collectorforkubernetes allows deploying collectord on non-master nodes.
  • DaemonSet collectorforkubernetes-master allows deploying collectord on master nodes.
  • Deployment collectorforkubernetes-addon is a single collectord instance that forwards data from the entire cluster.

Read the comments in the collectorforkubernetes.yaml file to get more detailed information about all configurations and sources of logs and metrics.

Collectord configuration

ConfigMap collectorforkubernetes delivers configuration files for collectord. These are ini files, where you can find all the default values.

Values can be overridden using environment values with the format as specified below

COLLECTOR__{ANY_NAME}={section}__{key}={value}

Configurations with environment variables are the simplest way to explore and debug quickly, but we recommend writing your configuration file based on the defaults provided with collectorforkubernetes.yaml.

Using secrets to manage configurations

You can use Kubernetes secrets and map them as environment variables to override configurations for collectord.

As an example, we will show how you can configure the HTTP Event Collector and License with secrets.

First, make sure that the collectorforkubernetes namespace already exists. If it does not exist, you need to create it.

kubectl create namespace collectorforkubernetes

kubectl create secret generic collectorforkubernetes \
    --namespace collectorforkubernetes \
    --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
    --from-literal=license="general__license="

In our YAML manifest, find the configuration of environment variables for each Deployment type (2 DaemonSets and 1 Deployment) and add the following information to the environment variables

env:
- name: COLLECTOR__SPLUNK_TOKEN
  valueFrom:
    secretKeyRef:
      name: collectorforkubernetes
      key: splunk-token
- name: COLLECTOR__LICENSE
  valueFrom:
    secretKeyRef:
      name: collectorforkubernetes
      key: license

Apply the manifest by following the installation instructions.

Attaching EC2 Metadata

You can include EC2 metadata with the forwarded data (logs and metrics) by specifying desired field name and path from Instance Metadata and User Data.

# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
# ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
# ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type

As an example, you can modify the YAML file and include two fields ec2_instance_id and ec2_instance_type

[general]
...
ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
...

Placeholders in indexes and sources

You can apply dynamic index names in the configurations to forward logs or stats to a specific index based on the meta fields. For example, you can define an index as:

[input.files]

index = oc_{{kubernetes_namespace}}

Similarly, you can change the source of all forwarded logs like:

[input.files]

source = /{{kubernetes_namespace}}/{{::coalesce(kubernetes_daemonset_name, kubernetes_deployment_name, kubernetes_statefulset_name, kubernetes_cronjob_name, kubernetes_job_name, kubernetes_replicaset_name, kubernetes_pod_name)}}/{{kubernetes_pod_name}}/{{kubernetes_container_name}}

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.