Monitoring Kubernetes - Release History

5.15.301 - 2020-06-24

Collectord updates:

• Bug fix: verify command broken for addon pod

5.15.300 - 2020-06-01

Requires collectorforkubernetes version 5.15.300 or above

• Events dashboard: filters depend on selection of cluster and node labels
• Support for Kubernetes 1.18+
• Improvement for alert "Cluster Warning: high number of errors to Kubernetes API" (only alert on 5xx errors)
• Bug fix: node events aren't visible in Events tab

Collectord updates:

• Support for annotations to add custom user fields to data
• Support for blacklisting and whitelisting Prometheus metrics (significally reducing the indexing cost of data)
• Verify command improvements - verify proper configurations for cgroup (memory/memory.use_hierarchy is 1)
• Bug fix: fix bug in prometheus metrics parser, empty fields can be filled with previous fields
• Bug fix: occasionally addon can report warnings about trying to delete expired keys from ack db
• Bug fix: better handle of connections to metrics endpoints exported in Prometheus format
• Bug fix: http connections improvements for when Splunk is unresponsive
• Bug fix: broken diag

5.14.284 - 2020-03-23

Collectord updates:

• New annotation to configure whitelist pattern for log messages
• Allow to override Kubernetes service URL
• Bug fix: panic in output for addon
• Bug fix: performance and memory usage improvement for ack db

5.14.280 - 2020-01-27

Requires collectorforkubernetes version 5.14.280 or above

• Logs dashboard: filters depend on selection
• Overview dashboard: namespace counter for list of projects

Collectord updates:

• Support templates in the index, source and sourcetype
• Allow to exclude indexed fields when forwarding to Splunk
• Support annotation for stats interval for containers
• Support containerd runtime
• Bug fix: verify command can show incorrect error about verifying journald input
• Bug fix: index on namespace should set index for application logs
• Bug fix: warning about not being able to retrieve node information

5.12.273 - 2019-11-18

Collectord updates:

• Bug fix: panic in application logs discovering for PVC volumes

5.12.272 - 2019-11-08

Collectord updates:

• Bug fix: in case when the rotated files are reusing FileID/DevID Collectord stops forwarding rotated files

5.12.271 - 2019-11-07

Requires collectorfordocker version 5.12.271 or above

• Improvements for the macros for backward compatibility

Collectord updates:

• Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
• Bug fix: reduce warnings failed to get the new event in pipeline - submitted
• Stability improvements

5.12.270 - 2019-10-22

Requires collectorforkubernetes version 5.12.270 or above

• Compact metrics (pre-calculated on Collectord side)
• Switched stats for host and cgroup in different macros
• Use base macro for alerts
• Improved command extraction for exec in Audit Logs
• Add cluster name in the alert results

Collectord updates:

• Watch namespaces and workloads for changes
• Global configurations with Custom Resources and selectors
• Describe command to see applied annotations for pods
• Bug fix: panic when pipe join configuration is removed
• Bug fix: panic when proc stats is enabled and cgroup stats is disabled
• Bug fix: support ProxyBasicAuthorization for license server checks
• Bug fix: Fix for collecting first sample (can show high CPU usage for first sample)
• Bug fix: if list of URLs is used for Splunk output, the empty URL is still required
• Beta: dynamic index, source and sourcetype names based on the metafields
• Beta: cluster diagnostics with one rule: node entropy

5.11.265 - 2020-06-24

Collectord updates:

• Bug fix: memory improvement for large ackdb files

5.11.264 - 2019-11-08

Collectord updates:

• Bug fix: in case when the rotated files are reusing FileID/DevID Collectord stops forwarding rotated files

5.11.261 - 2019-09-13

Collectord update:

• Bug fix: improves discovery for the PVC volumes
• Bug fix: delay loading for the PVC volumes
• Bug fix: improves logging for the directory walker

5.11.260 - 2019-09-09

Requires collectorforkubernetes version 5.11.260 or above

• GPU Monitoring (NVIDIA)

Collectord updates:

• Support for PVC volumes for application logs
• Bug fix: small memory leak in addon
• Bug fix: duplicate events then pipeline is getting throttled
• Bug fix: don't use throttling for devnull output
• Bug fix: better recovery for ack db corruption
• Bug fix: crash on journald input initialization when ack db is corrupted
• Bug fix: annotations joinmultiline requires joinpartial
• Bug fix: configurations for stdout only with annotations can crash collectord
• Set events = 50 by default for Splunk output batches

5.10.255 - 2019-11-20

Collectord updates:

• Bug fix: better recovery for ack db corruption
• Bug fix: crash on journald input initialization when ack db is corrupted

5.10.253 - 2019-07-31

Collectord update:

• Bug fix: collectord can pick up compressed json logs (*.gz)
• Bug fix: too verbose warnings from the docker watcher about retries

5.10.252 - 2019-07-24

Collectord update:

• Support for configuring the thruput (general and with annotations for container logs)
• Support for configuring too old or too new events (general and with annotations for container logs)

5.10.251 - 2019-06-20

Collectord update:

• Ability to configure Acknowledgement database for collectord.

5.10.250 - 2019-06-18

Requires collectorforkubernetes version 5.10.250 or above

• Security dashboard: Access: access to host via ssh, sudo, exec commands, failed access
• Security dashboard: Audit (users and namespaces)
• Security dashboard: Network (traffic)
• Security dashboard: Network (connections)
• Security dashboard: Objects (pods) - review pods with host network, age of pods, image pull policy, attached host paths, security context and restart policies
• Review dashboard: Clusters (allocations and usage)
• Cluster field filters
• Base macro for overriding macros for other macros

Collectord updates:

• Support for volatile and persistent journald storage with default configuration
• Updated YAML configuration to include most common resources
• Better support for overriding sourcetype, that does not require to update the Splunk macros
• Bug fix: rarely when collectord fails to post to HEC it can panic
• Bug fix: better support for Kubernetes 1.14 and CRI-O storage
• Bug fix: space characters in index annotations can break the pipeline

5.9.244 - 2019-05-20

Collectord update:

• Bug fix: support for CRI-O in Kubernetes 1.14

5.9.240 - 2019-05-14

Requires collectorforkubernetes version 5.9.240 or above

• Visual improvements on the graphs for the number of logs and events
• New alerts for the CPU and Memory reservation

Collectord updates:

• Support for multiple Splunk destinations (outputs)
• Support subdomains for annotations (to deploy multiple collectord instances)
• Support for streaming objects from Kubernetes API to Splunk
• Bug fix: journald input keeps fd open to the rotated files
• Bug fix: fix in the annotation parser for the interval annotations
• Bug fix: fix splunk url selection configuration for multiple splunk URLs

5.8.231 - 2019-04-25

• Bug fix: Collectord usage report shows trial licenses for all instances

5.8.230 - 2019-04-22

Requires collectorforkubernetes version 5.8.230 or above

• Use multiselect filters for most dashboards and filters with possibility to input custom filters.
• Reduce dedup usage to improve performance on dashboards.
• Add critical pod annotations for Kubernetes ...1.13, and priority class for Kubernetes 1.14...
• Fix: statefulset dashboard does not show data with filters.
• Add graph of number of pods per namespace on Overview dashboard.

Collectord updates:

• Bug fix: clogging collectord output with errors when incorrect index is used.
• Bug fix: short lived containers can results in duplicating logs.
• Bug fix: clogging collectord output with warnings when kernel reports incorrect VmRss size.
• Bug fix: annotations cannot override timestamp location for fields extraction.
• Bug fix: verify command reports Journald input in incorrect place.
• Better support for cgroup symlinks, automatically discover correct location.

5.7.220 - 2019-03-18

Requires collectorforkubernetes version 5.7.220 or above

• Review savedsearches/alerts to support indexing delay (start searches from 2 minutes behind) and run them in more random time.
• Workload dashboard - change CPU (of host) in table to real CPU
• Fixed single value memory panel on host dashboard (missed span)
• Use SEGMENTATION=none for stats events to use less disk space (needs to me moved to indexers)

Collectord updates:

• Support hostname formatting with environment variables in configuration
• New rotated file logic uses less file descriptors and frees rotated files quicker
• Allow to specify a default sampling value for container logs
• Reimplemented shutdown sequence to stop collectord faster
• Allow to override sampling percent with annotations
• New Input: journald

5.6.213 - 2019-03-03

• Collectord: Fix panic, when collectord does not have access to docker socket, and information about this container does not exist on the disk.

5.6.212 - 2019-02-19

Requires collectorforkubernetes version 5.6.212 or above

• New: Alert: high CPU usage on the host.
• Fixed: Splunk usage dashboard - charts do not show the data, when the used indexed aren't searchable by default.
• New: Support Dark theme.
• New: Free text search in Logs dashboard.
• New: Add auto-refresh options to the dashboard.
• Fixed: Revisited CPU limits and requests for Pods and Containers.
• New: add CPU Max, Memory Max and Project/Namespace labels to the Review-Namespaces dashboard.
• Fixed: Show deleted events

Collectord updates:

• Fixed: auto-recovery from the corrupted write-ahead-log in acknowledgment database.
• New: support sampling (random and hash-based) for container/application and host logs.
• New: when running multiple collectord on one host (with different output) - count that as one licensed host, change InstanceID format.
• Fixed: when container is scheduled with remove flag lock the file till collectord processes it completely.
• Fixed: collectord reports rare warning about unparsable uint64 max value from proc filesystem.
• Fixed: collectord reports rare warning about unparsable line from proc/io files.
• New: allow to include annotations in the forwarding data.
• Fixed: if collectord cannot access to the API - report the warning less often
• Fixed: do not report docker warnings for verify command, if there is no container scheduled outside of the Kubernetes.
• New: splunk output - allow to limit the output batch by the number of events in payload.
• Fixed: attach namespace labels to the forwarded logs.
• Fixed: attach openshift_namespace field to the events.

5.5.205 - 2019-01-25

• Collectord fix: collectord could stop sending container file logs when the original file has been truncated (using the same Node ID as previously used log file).

5.5.203 - 2019-01-25

• Collectord fix: collectord could send an empty X-Splunk-Request-Channel header to Splunk.

5.5.202 - 2019-01-24

Requires collectorforkubernetes version 5.5.202

• New: Dashboard Review -> namespaces. Review allocations and requests for namespaces and pods.
• Fixed: kubernetes_stats_cpu_request_percent - is divided by the number of CPU.

Collectord updates:

• Fixed: Interval 0 in prometheus input can crash the collectord.
• Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from finding the files in the volume.

5.4.201 - 2018-12-19

Requires collectorforkubernetes version 5.4.201 or above

• Fixed: Alerts for licenses issued with AWS Subscriptions

Collectord updates:

• Fixed: Better handling rotated files (less open fd)
• Fixed: Events input can hang in the err loop.

5.4 - 2018-12-17

Requires collectorforkubernetes version 5.4 or above

• New: CoreDNS dashboard.
• New: CoreDNS alerts.
• Improved: etcd metrics representation for bucket values.
• Compatibility update for collectord 5.4.

Collectord updates:

• New: Attach EC2 metadata fields
• New: Basic Auth for Proxy (License Server and Splunk)
• Fixed: Collectord verify reports CRI-O as unsupported runtime.
• Fixed: Rare crash on Prometheus metrics definition.
• Fixed: Better handling of acknowledgment database corruption.
• Fixed: When handling incorrect indexes, collectord can send index with empty string, that Splunk recognize as incorrect index

5.3 - 2018-11-19

Requires collectorforkubernetes version 5.3 or above

• Fixed: Improved Workload dashboard. Allows to filter by namespace, see all Pods in a specific namespace, filter by workload label.
• New: Alert for showing when Collectord reports errors in Processing pipelines (as an example if it failed to extract fields).
• New: Alert for showing when Collectord reports warnings.
• Fixed: Add node labels filter to Storage Dashboard and Control Plane Dashboards.
• New: Alert if lag in the indexing of the data.
• New: Splunk Usage (License usage, number of events) report under Setup.
• Fixed: adjusted high amount of errors to Kubernetes API dashboard to make it less verbose.
• Fixed: misprint in the search for showing alerts
• Fixed: lookup with alerts causing very often replication activities on SHC
• Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

Collectord updates:

• Fixed: high memory usage with Gzip compression enabled (reduced memory usage).
• New: Allow to disable pipe.join with annotations.
• Fixed: In high amount of logs (10,000 events per second) Collectord can read lines not in full, that can break JSON logs.
• Fixed: When collectord writes a Warning that it failed to post to Splunk, it will write a Success message after retry.
• New: Allow to hash sensitive data with annotations.
• Fixed: Group network socket tables to reduce the amount of forwarded data (4 times reducing the amount of data)
• Fixed: Identify when glob and match pattern require recursive directory traversal.
• Fixed: Make it possible to add annotations for the specific containers inside of the the same Pods.
• New: Annotation for complete disabling of the handling and forwarding logs for containers.
• Fixed: Performance improvements for CRI-O logs.
• Fixed: Collectord showed few Debug messages on start.
• Fixed: Performance improvements for log forwarding (up to 35% in high amount of logs).
• Fixed: reduce duplication of the Kubernetes events, forwarded to Splunk.
• Fixed: Do not generate a WARN when API Server results in 404. Usually this caused by the owner object being deleted.
• Fixed: Failed to parse proc name from the stat file with the not paired parentheses.

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above

• New: Review/Storage dashboard based on storage metrics and PVC metrics.
• New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
• Fixed: Performance improvements

Collector updates:

• New: runtime storage metrics (usage, available, inodes)
• New: image is built on top of SCRATCH image.
• New: verify and diag commands for troubleshooting.
• New: support /dev/null output for logs
• New: override source/sourcetype and index base on regexp pattern for container logs.
• Fixed: do not send empty docker_labels
• New: support docker JSON tags and labels
• Fixed: allowing a new license to unblock collector with the expired license.
• Fixed: Prometheus parser fails to parse metrics with labels that end with a comma.
• Fixed: Performance improvements
• New: Prometheus parser supports basic authentication
• Fixed: Workaround for a bug in HTTP Event Collector, that can return an incorrect index of failed event
• New: Prometheus autodiscover support host network
• Fixed: remove node info and limit metadata from logs
• Fixed: documentation / default configuration update - mount `/etc/localtime to allow collector to use host tz (when not UTC)
• Fixed: documentation / default configuration update - use dnsPolicy: ClusterFirstWithHostNet for pods mounted on host network

5.1 - 2018-09-17

Requires collectorforkubernetes version 5.1 or above

• New: Network metrics (MB, Packets, Drops and Errors) for host and containers.
• New: Network socket tables (list of port that containers and hosts are listen on, connections to external resources).
• New: Network review dashboard to see the list of connection to public services and in private network.
• Improvement: Replace python-based lookup with macro written with eval.
• Improvement: Visual improvement for showing when the object was Last Seen (highlighting and showing minutes ago).
• New: discovering Prometheus metrics in Pods with annotations.
• New: attaching pod metadata to metrics collected from prometheus metrics exposed from pods.
• Improvement: Changed source of proc stats to proc root filesystem, to keep minimum list of unique sources.
• New: Support for Splunk multi-threads outputs (for forwarding more than 3000 events per second).
• Improvement: Performance improvements for Prometheus parsing.
• Improvement: Reduce amount of metrics forwarded with proc_stats by excluding system threads.
• Improvement: Configuration for gzip compression.
• Improvement: Calculate checksums for first bytes of files, to better identify new files with reused iNode.
• Bug: Process metrics could be collected 2 times.

5.0 - 2018-09-03

Requires collectorforkubernetes version 5.0 or above

• New dashboard: Events
• Added events panel to the Workload and Pod dashboards.
• Labels on Workload and Hosts dashboards.
• Auto-discover and forward Application logs from host mounts or local volumes.
• Annotations for containers to change per container configurations (index, source, join rules, replaces and more).
• Escaping terminal sequences from container logs.
• Redirecting logs to /dev/null for specific patterns.
• Replace patterns in container and application logs (hiding sensitive or not important information).
• Support for extracting fields from the container logs, including timestamps.
• Include Memory and CPU limits for container lists.
• Visual updates for the panels, highlighting high CPU and Memory usages
• Filter cgroup stats, forward only container and host metrics.
• Support for multiple Splunk HTTP Event Collector endpoints (support fail-over and load-balancing).
• Handle HTTP Event Collector errors with the incorrect index. Multiple options to redirect to default index, drop or wait.
• Add retry logic to license client to reduce amount of false positive warnings.
• Add HTTP read timeouts (handle gateway timeouts, 504).
• Fixed: fail to parse the latest line in the JSON log.
• Better error handling incorrect configurations.
• Deprecating Join rules in favour of annotations.
• Support for HTTP Event Collector client certificates.
• Support CRI-O runtime.
• Fixed: limit directory walkers for depth (fixing issues when directory has a mount to itself)
• Fixed: add a limit of the maximum line size that collector can read at once (defaults to 1Mb).
• Fixed: acknowledgement database stores now NodeID, DevID and a parent folder identifier. That way if NodeID is going to be reused right away - we will identify this file as a new one, if it is in different location.
• Change: docker_stream field has been renamed to stream for compatibility with other container runtime.
• Change: prometheus metrics has default sourcetype=kubernetes_prometheus (macro supports backward compatibility)

Upgrade from version 4 to 5

4.0.24 - 2018-05-05

Requires collectorforkubernetes version 4.0 or above

• New dashboard: Cluster/Audit
• New dashboard: Cluster/Kubernetes API Server
• New dashboard: Cluster/Kubelet
• New dashboard: Cluster/etcd
• New dashboard: Cluster/Scheduler
• New dashboard: Cluster/Controller Manager.
• Include image name, when list containers.
• Added syslog component to the list of host logs.
• Fixed: Include Daemon Set on Overview dashboard, list of namespaces.

Collector updates (4.0.171):

• Collecting metrics from Prometheus format.
• Add HTTP read timeouts (handle gateway timeouts, 504).
• Correctly parse HTTP Event Responses when one of few events fail to be indexed (as an example, wrong index).
• Performance optimizations.
• Optimize payloads for higher write throughput.
• Fixed: reduce the number of calls to Kubernetes API Server.
• Fixed: fail to parse the latest line in the JSON log.
• Better error handling incorrect configurations.
• Failed to parse memory limits (Failed to parse memory=000k for the container).
• Collecting Kubernetes events from the cluster once by using collector addon.

collectorforkubernetes 4.0.172

• Fixed: Messages "WARN ... proc.go:441: Unparsable line from /rootfs/proc/X/status" caused by new Linux kernel that reports empty line in proc file system.
• Fixed: Incorrectly parsed Limits for the Kubernetes pods. 5m and 500m both results as 0.500.

collectorforkubernetes 4.0.173

• Fixed: significant memory usage with the events larger than 512Kb, caused by Splunk issue SPL-156315 (incapable to parse events larger 512Kb, regression in 7.x).

collectorforkubernetes 4.0.174.180730

• Show the index name in the output, when Splunk reports incorrect index.

Upgrade from version 3 to 4

3.0.23 - 2018-02-17

Requires collectorforkubernetes version 3.0 or above

• Bug: Memory view on workflow dashboard had a max limit set to 100.
• Bug: Events view on overview dashboard had a max limit set to 100.

3.0.22 - 2018-02-07

Requires collectorforkubernetes version 3.0 or above

• Added support for containers deployed without Kubernetes.
• Added CPU Quota, CPU Shares, Throttled and Memory Limit and Request Overlays on Container and Pod Dashboards.
• Indexing Kubernetes events in sourcetype kubernetes_events
• Performance improvement on Dashboards by combining multiple charts using one common search.
• New "Review/Allocatable Resources" dashboard to track limits and requests for CPU and Memory.
• New "Review/Privileged containers and enabled capabilities" dashboard to list all privileged containers and enabled security capabilities for containers.
• New Overview dashboard to easy navigate within the application.
• New Aggregated metrics dashboard for specific Workload.
• Fixed bug on Process Dashboard, some charts did not filter by host.
• "Setup: Collectors" now supports collectorforkubernetes images distributed via private registries.
• "Overview: Process" dashboard did not use Span token for timechart dashboards.
• "Top: Containers" fixed incorrect memory usage (showed double size)
• Added alerts in application for notification about outdated collector versions and expired licenses for collector.
• Hide Wait Read/Write IO panels, when this data is not available.
• In process Dashboard show VmRSS with RssAnon, RssFile, and RssShmem.

Collector updates:

• Support for Splunk indexing acknowledgment.
• Watching for Kubernetes/OpenShift events.
• HTTP Proxy support for License server and Splunk output.
• Allow to configure destination indices for different types of data in collector configuration (stats, logs, host logs, proc stats and events).
• Handling responses from HTTP Event Collector to skip invalid events (will be logged).
• If container is running, but Kubernetes does not provide metadata, allow to wait for metadata.
• Collect security capabilities and uid/gid.
• For Kubernetes/OpenShift environments recognize containers scheduled outside of Pods and load metadata directly from docker.
• Support for custom labels, specified with collector configuration.
• Support OpenShift/Kubernetes annotations "collectord.io/..." to configure destination indices, sourcetypes and sources for pods, workloads and namespaces.
• Support for partial logs without join rules.
• Bug. Use local timezone by default for local syslog files.
• Bug. Fix small memory leak on deleted containers.
• Bug. When collector is failing to send data to Splunk, impossible to stop collector with terminate.

Upgrade from version 2 to 3

2.1 - 2017-10-22

Requires collectorforkubernetes version 2.1.59.171210 or above

• Implemented collectors dashboard to track number of collectors, their versions and used licenses.
• Fallback to the process IO statistics when blkio is not available.
• Fix IO statistic graphs, showed average, when sum should be used.
• Fields extraction support for nginx ingress 0.9 and above.
• collector* - Improved resistance for storage failures.
• collector* - License checks reporting.
• collector* - Better support for openshift environment (default configuration).

2.0 - 2017-10-22

Requires collectorforkubernetes version 2.0.37.171023 or above

• Better labels support in Dashboards. Collector has a breaking feature, replacing format for labels from kubernetes_node_labels_LABEL1=VALUE1 to kubernetes_node_labels=[LABEL1=VALUE1,LABEL2=VALUE2].
• Process level metrics.
• Uptime for hosts and processes.
• Fields extraction for kubernetes controller manager and scheduler.
• Fields extraction and support in dashboards for main kubernetes components (setup host logs collection with collector).
• New top-like dashboards allow to monitor Hosts/Pods/Containers/Processes in real-time.
• Rewritten Kubernetes Objects Dashboards with support of Events and Labels.
• Improved dashboards navigation.
• Support for host logs.
• Other bugs and improvements based on user feedback.

Links

• Documentation
• FAQ and the common questions
• License agreement
• Pricing
• Contact