You are looking at documentation for an older release. See the current release documentation.
Monitoring Kubernetes Configuration
Created Kubernetes Objects
Configuration file collectorforkubernetes.yaml creates several Kubernetes Objects.
ClusterRolecollectorforkuberneteswith limited capabilities toget,listandwatchdeployed objects. Collector uses this information to enrich logs and stats with kubernetes specific metadata.ServiceAccountcollectorforkubernetesis used to connect to Kubernetes API.ClusterRoleBindingcollectorforkubernetesto bind service account to cluster role.ConfigMapcollectorforkubernetesdelivers configuration file for collector.DaemonSetcollectorforkubernetesallows to deploy collector on every node, including master node.
Read commentaries in collectorforkubernetes.yaml file to get more deep details on all configurations and source of the logs and metrics.
Collector configuration
ConfigMap collectorforkubernetes delivers configuration file for collector.
This is an ini file, where all the configuration values are commented out.
All the commented out values are default values.
Values can be overridden using environment values with the format as specified below
COLLECTOR__{ANY_NAME}={section}__{key}={value}
Configurations with environment variables are the simplest way to explore and debug quickly, but we recommend to write your configuration file based on the default provided with collectorforkubernetes.yaml.
Join Rules
By default collector joins all messages with previous if they start with spaces. Below you can find how to specify a custom rule on the example of java application.
If this is a sample of the application logs.
[2017-09-04T06:28:05,664][WARN ][MyComponent] java.security.AccessControlException: access denied at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131] at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131] [2017-09-04T06:28:05,664][WARN ][MyComponent] another message
You can specify the join rules, where you configure that you want to match all containers with the name that contains my_app
in their name, and pattern for the new message should match regex ^\[\d{4}-.
[pipe.join::my_app] matchRegex.kubernetes_container_name = .+my_app.+ patternRegex = ^\[\d{4}-
Cluster labels
Our dashboards allows you to filter nodes based on the node labels.
If you have two clusters prod and dev, each cluster has master1, node1 and node2 nodes you can
apply labels to every node with kubectl.
As an example, in the dev cluster for the node master you can append label example.com/cluster: dev.
$ kubectl edit nodes/master1
Find labels list and append new label.
1 2 3 4 5 6 | labels: beta.kubernetes.io/arch: amd64 beta.kubernetes.io/os: linux kubernetes.io/hostname: master1 node-role.kubernetes.io/master: "" example.com/cluster: dev |
If you do that for all of the nodes in all of your clusters, you will be able to use these labels on most of the
dashboards of our applications. With the given example, you will be able to filter by labels example.com/cluster=dev
and example.com/cluster=prod.
Our collector reads node labels only at the start. To apply this change tothe collector you need to restart it.
Comparing with other solutions
Comparing with Fluentd
- Logs are enriched with more Kubernetes metadata. Allowing to filter not only by pod names but also by ownership workflow.
- Tiny image, tiny process. Uses 4 times less Memory on our tests.
- Support for multi-line events. Specify rules for containers on how you identify new events.
- Collects metrics for all running containers. Allows you to correlate metrics with logs.
- Collects process metrics.
- Flexible source pattern allows you to specify field extraction rules.
- Pre-built application
collectorforkubernetes.yaml
Download the latest configuration file from the Configuration Reference page.
