Monitoring Docker - Release History

5.16.363 - 2021-05-26

Collectord updates:

• Upgrade go runtime to 1.16.3
• Bug fix: fix verbose logging for docker watcher with messages "failed to get next event"
• Bug fix: Verify command fails on Containerd runtime
• Bug fix: DefaultIdleConnTimeout is ignored for HTTP clients

5.16.361 - 2021-03-16

Requires collectorfordocker version 5.16.350 or above

• Bug fix: use correct units for Memory and Storage (MiB, MB, Mb)
• Bug fix: Collectord metrics request time shows the summary on the period, not the individual request times

Collectord updates:

• ARM64 image
• Upgrade to Go 1.16.2
• Bug fix: precise time to Splunk HEC, sending with milliseconds instead of nanoseconds (which are incorrectly ronded by HEC)
• Bug fix: first sample of the container can record above 100% of the CPU usage, as the values are pretty low
• Bug fix: verify command does not respect glob patterns for Prometheus inputs (certs, tokens)
• Bug fix: trim spaces in token value for Prometheus inputs

5.16.353 - 2021-02-11

Collectord updates:

• Bug fix: collectord can report parse int errors on the stderr
• Upgrade go runtime to 1.15.8

5.16.351 - 2021-01-04

Collectord updates:

• Bug fix: host file inputs can raise a fatal error: concurrent map writes

5.16.350 - 2020-12-14

Requires collectorfordocker version 5.16.350 or above

• New dashboard: Collectord metrics

Collectord updates:

• Annotations for collecting prometheus metrics: authorization keys and CAName for SSL certificates
• Improvement for DNS resolutions of Splunk output FQDN
• Export internal collectord metrics in Prometheus format
• Forwarding internal collectord metrics to Splunk
• For the watch objects inputs being able to hide management fields
• In the diag include all open file descriptors
• Upgrade go runtime to 1.14.13
• Remove \0 symbol from the labels values in the prometheus metrics
• Allow to filter host logs with blacklist and whitelist
• Bug fix: less verbose warnings about not being able to load resources from API server
• Bug fix: performance improvements for Ack DB
• Bug fix: custom prometheus metrics forwarded by Collectord do not include cluster field or custom user fields
• Bug fix: addon pod terminates faster
• Bug fix: verify command trying to post to all outputs with all indexes specified in the configuration
• Bug fix: crash in AckDB
• Bug fix: input system stats does not recognize ouputs specified for the host and cgroup
• Bug fix: verify command runs recursively all the time for host logs even when recursively is set to false

5.15.305 - 2021-01-04

Collectord updates:

• Upgrade go runtime to 1.14.13
• Bug fix: host file inputs can raise a fatal error: concurrent map writes

5.15.304 - 2020-09-30

Collectord updates:

• Bug fix: cannot disable forwarding of the api resources

5.15.303 - 2020-08-12

Collectord updates:

• Upgrade golang to 1.14.7 to fix the hang in runtime

5.15.301 - 2020-06-24

Collectord updates:

• Bug fix: verify command broken for addon pod

5.15.300 - 2020-06-01

Requires collectorforkubernetes version 5.15.300 or above

Collectord updates:

• Support for annotations to add custom user fields to data
• Support for blacklisting and whitelisting Prometheus metrics (significally reducing the indexing cost of data)
• Verify command improvements - verify proper configurations for cgroup (memory/memory.use_hierarchy is 1)
• Bug fix: fix bug in prometheus metrics parser, empty fields can be filled with previous fields
• Bug fix: occasionally addon can report warnings about trying to delete expired keys from ack db
• Bug fix: better handle of connections to metrics endpoints exported in Prometheus format
• Bug fix: http connections improvements for when Splunk is unresponsive
• Bug fix: broken diag

5.14.285 - 2020-08-12

Collectord updates:

• Upgrade golang to 1.14.7 to fix the hang in runtime

5.14.284 - 2020-03-23

Collectord updates:

• New annotation to configure whitelist pattern for log messages
• Allow to override Kubernetes service URL
• Bug fix: panic in output for addon
• Bug fix: performance and memory usage improvement for ack db

5.14.280 - 2020-01-27

Requires collectorforkubernetes version 5.14.280 or above

• Logs dashboard: filters depend on selection

Collectord updates:

• Support templates in the index, source and sourcetype
• Allow to exclude indexed fields when forwarding to Splunk
• Support annotation for stats interval for containers
• Bug fix: verify command can show incorrect error about verifying journald input
• Bug fix: index on namespace should set index for application logs

5.12.273 - 2019-12-16

Requires collectorfordocker version 5.12.272 or above

• Bug fix: Swarm Services dashboard compatibility with the new format of metrics

5.12.272 - 2019-11-08

Collectord updates:

• Bug fix: in case when the rotated files are reusing FileID/DevID Collectord stops forwarding rotated files

5.12.271 - 2019-11-07

Requires collectorfordocker version 5.12.271 or above

• Improvements for the macros for backward compatibility
• Bug fix: Swarm Services dashboard could not filter based on the stack name

Collectord updates:

• Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
• Bug fix: reduce warnings failed to get the new event in pipeline - submitted
• Stability improvements

5.12.270 - 2019-10-22

Requires collectorfordocker version 5.12.27- or above

• Compact metrics (pre-calculated on Collectord side)
• Switched stats for host and cgroup in different macros
• Containers count for hosts on Hosts and Host dashboards
• Improve performance of the search in Docker Services and ECS Services dashboards
• Use base macro for alerts
• Add cluster name in the alert results

Collectord updates:

• Describe command to see applied annotations for containers
• Bug fix: panic when pipe join configuration is removed
• Bug fix: panic when proc stats is enabled and cgroup stats is disabled
• Bug fix: support ProxyBasicAuthorization for license server checks
• Bug fix: Fix for collecting first sample (can show high CPU usage for first sample)
• Beta: dynamic index, source and sourcetype names based on the metafields
• Beta: cluster diagnostics with one rule: node entropy

5.11.266 - 2020-10-15

Collectord updates:

• Upgrade golang to 1.14.10 to fix the hang in runtime

5.11.265 - 2020-06-24

Collectord updates:

• Bug fix: memory improvement for large ackdb files

5.11.264 - 2019-11-08

Collectord updates:

• Bug fix: in case when the rotated files are reusing FileID/DevID Collectord stops forwarding rotated files

5.11.261 - 2019-09-13

Collectord update:

• Bug fix: improves logging for the directory walker

5.11.260 - 2019-09-09

Requires collectorfordocker version 5.11.260 or above

• Bug fix: changing source type does not allow to use Swarm and ECS services dashboards
• Bug fix: improving usability of Swarm and ECS services dashboards for large deployments

Collectord updates:

• Bug fix: duplicate events then pipeline is getting throttled
• Bug fix: don't use throttling for devnull output
• Bug fix: better recovery for ack db corruption
• Bug fix: crash on journald input initialization when ack db is corrupted
• Bug fix: annotations joinmultiline requires joinpartial
• Bug fix: configurations for stdout only with annotations can crash collectord
• Set events = 50 by default for Splunk output batches

5.10.255 - 2019-11-20

Collectord updates:

• Bug fix: better recovery for ack db corruption
• Bug fix: crash on journald input initialization when ack db is corrupted

5.10.253 - 2019-07-31

Collectord update:

• Bug fix: collectord can pick up compressed json logs (*.gz)
• Bug fix: too verbose warnings from the docker watcher about retries

5.10.252 - 2019-07-24

Collectord update:

• Support for configuring the thruput (general and with annotations for container logs)
• Support for configuring too old or too new events (general and with annotations for container logs)

5.10.251 - 2019-06-20

Collectord update:

• Ability to configure Acknowledgement database for collectord.

5.10.250 - 2019-06-18

Requires collectorforopenshift version 5.10.250 or above

• Cluster field filters
• Base macro for overriding macros for other macros

Collectord updates:

• Support for volatile and persistent journald storage with default configuration
• Updated YAML configuration to include most common resources
• Better support for overriding sourcetype, that does not require to update the Splunk macros
• Bug fix: rarely when collectord fails to post to HEC it can panic
• Bug fix: better support for OpenShift 4.x and CRI-O storage
• Bug fix: space characters in index annotations can break the pipeline

5.9.240 - 2019-05-14

Requires collectorfordocker version 5.9.240 or above

• Visual improvements on the graphs for the number of logs and events

Collectord updates:

• Support for multiple Splunk destinations (outputs)
• Support subdomains for annotations (to deploy multiple collectord instances)
• Bug fix: journald input keeps fd open to the rotated files
• Bug fix: fix in the annotation parser for the interval annotations
• Bug fix: fix splunk url selection configuration for multiple splunk URLs

5.8.231 - 2019-04-25

• Bug fix: Collectord usage report shows trial licenses for all instances

5.8.230 - 2019-04-22

Requires collectorfordocker version 5.8.230 or above

• Bug fix: Swarm dashboard does not render containers, when namespace field is not available.
• Use multiselect filters for most dashboards and filters with possibility to input custom filters.
• Reduce dedup usage to improve performance on dashboards.

Collectord updates:

• Bug fix: clogging collectord output with errors when incorrect index is used.
• Bug fix: short lived containers can results in duplicating logs.
• Bug fix: clogging collectord output with warnings when kernel reports incorrect VmRss size.
• Bug fix: annotations cannot override timestamp location for fields extraction.
• Bug fix: verify command reports Journald input in incorrect place.
• Better support for cgroup symlinks, automatically discover correct location.

5.7.220 - 2019-03-18

Requires collectorfordocker version 5.7.220 or above

• Review savedsearches/alerts to support indexing delay (start searches from 2 minutes behind) and run them in more random time.
• Fixed single value memory panel on host dashboard (missed span)
• Use SEGMENTATION=none for stats events to use less disk space (needs to me moved to indexers)

Collectord updates:

• Support hostname formatting with environment variables in configuration
• New rotated file logic uses less file descriptors and frees rotated files quicker
• Allow to specify a default sampling value for container logs
• Reimplemented shutdown sequence to stop collectord faster
• Allow to override sampling percent with annotations
• New Input: journald

5.6.213 - 2019-03-03

• Collectord: Fix panic, when collectord does not have access to docker socket, and information about this container does not exist on the disk.

5.6.212 - 2019-02-19

Requires collectorfordocker version 5.6.212 or above

• New: Alert: high CPU usage on the host.
• Fixed: Splunk usage dashboard - charts do not show the data, when the used indexed aren't searchable by default.
• New: Support Dark theme.
• New: Free text search in Logs dashboard.
• New: Add auto-refresh options to the dashboard.
• Fixed: Revisited CPU limits and requests for Pods and Containers.

Collectord updates:

• Fixed: auto-recovery from the corrupted write-ahead-log in acknowledgment database.
• New: support sampling (random and hash-based) for container/application and host logs.
• New: when running multiple collectord on one host (with different output) - count that as one licensed host, change InstanceID format.
• Fixed: when container is scheduled with remove flag lock the file till collectord processes it completely.
• Fixed: collectord reports rare warning about unparsable uint64 max value from proc filesystem.
• Fixed: collectord reports rare warning about unparsable line from proc/io files.

5.5.205 - 2019-01-25

• Collectord fix: collectord could stop sending container file logs when the original file has been truncated (using the same Node ID as previously used log file).

5.5.203 - 2019-01-25

• Collectord fix: collectord could send an empty X-Splunk-Request-Channel header to Splunk.

5.5.202 - 2019-01-24

Requires collectorfordocker version 5.5.202 or above

• New: Dashboard Services -> AWS ECS. Review containers running as an ECS Service.
• New: Dashboard Services -> Swarm Services. Review containers running as a Swarm Service.

Collectord updates:

• Fixed: Interval 0 in prometheus input can crash the collectord.
• Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from finding the files in the volume.

5.4.201 - 2018-12-19

Requires collectorfordocker version 5.4.201 or above

• Fixed: Alerts for licenses issued with AWS Subscriptions

Collectord updates:

• Fixed: Better handling rotated files (less open fd)
• Fixed: Events input can hang in the err loop.

5.4 - 2018-12-17

Requires collectorfordocker version 5.4 or above

• Compatibility update for collectord 5.4.

Collectord updates:

• New: Attach EC2 metadata fields
• New: Basic Auth for Proxy (License Server and Splunk)
• Fixed: Collectord verify reports CRI-O as unsupported runtime.
• Fixed: Rare crash on Prometheus metrics definition.
• Fixed: Better handling of acknowledgment database corruption.
• Fixed: When handling incorrect indexes, collectord can send index with empty string, that Splunk recognize as incorrect index

5.3 - 2018-11-19

Requires collectorfordocker version 5.3 or above

• New: Alert for showing when Collectord reports errors in Processing pipelines (as an example if it failed to extract fields).
• New: Alert for showing when Collectord reports warnings.
• New: Alert if lag in the indexing of the data.
• New: Splunk Usage (License usage, number of events) report under Setup.
• Fixed: lookup with alerts causing very often replication activities on SHC

Collectord updates:

• Fixed: high memory usage with Gzip compression enabled (reduced memory usage).
• New: Allow to disable pipe.join with annotations.
• Fixed: In high amount of logs (10,000 events per second) Collectord can read lines not in full, that can break JSON logs.
• Fixed: When collectord writes a Warning that it failed to post to Splunk, it will write a Success message after retry.
• New: Allow to hash sensitive data with annotations.
• Fixed: Group network socket tables to reduce the amount of forwarded data (4 times reducing the amount of data)
• Fixed: Identify when glob and match pattern require recursive directory traversal.
• New: Annotation for complete disabling of the handling and forwarding logs for containers.
• Fixed: Collectord showed few Debug messages on start.
• Fixed: Performance improvements for log forwarding (up to 35% in high amount of logs).
• Fixed: reduce duplication of the Kubernetes events, forwarded to Splunk.
• Fixed: Support ECS cgroup matching with the default configuration.
• Fixed: Support docker daemon logs forwarding with the default configuration.
• Fixed: Failed to parse proc name from the stat file with the not paired parentheses.

5.2 - 2018-10-15

Requires collectorfordocker version 5.2 or above

• New: Review/Storage dashboard based on storage metrics.
• New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
• Performance improvements

Collector updates:

• New: runtime storage metrics (usage, available, inodes)
• New: image is built on top of SCRATCH image.
• New: verify and diag commands for troubleshooting.
• New: support /dev/null output for logs
• New: override source/sourcetype and index base on regexp pattern for container logs.
• Fixed: do not send empty docker_labels
• New: support docker JSON tags and labels
• Fixed: allowing a new license to unblock collector with the expired license.
• Fixed: Prometheus parser fails to parse metrics with labels that end with a comma.
• Fixed: Performance improvements
• New: Prometheus parser supports basic authentication
• Fixed: Workaround for a bug in HTTP Event Collector, that can return an incorrect index of a failed event

5.1 - 2018-09-17

Requires collectorfordocker version 5.1 or above

• New: Network metrics (MB, Packets, Drops and Errors) for host and containers.
• New: Network socket tables (list of port that containers and hosts are listen on, connections to external resources).
• New: Network review dashboard to see the list of connection to public services and in private network.
• Improvement: Replace python-based lookup with macro written with eval.
• Improvement: Visual improvement for showing when the object was Last Seen (highlighting and showing minutes ago).
• Improvement: Changed source of proc stats to proc root filesystem, to keep minimum list of unique sources.
• New: Support for Splunk multi-threads outputs (for forwarding more than 3000 events per second).
• Improvement: Performance improvements for Prometheus parsing.
• Improvement: Calculate checksums for first bytes of files, to better identify new files with reused iNode.
• Improvement: Reduce amount of metrics forwarded with proc_stats by excluding system threads.
• Improvement: Configuration for gzip compression.

5.0 - 2018-09-03

Requires collectorfordocker version 5.0 or above

• Auto-discover and forward Application logs from host mounts or local volumes.
• Annotations for containers to change per container configurations (index, source, join rules, replaces and more).
• Escaping terminal sequences from container logs.
• Redirecting logs to /dev/null for specific patterns.
• Replace patterns in container and application logs (hiding sensitive or not important information).
• Support for extracting fields from the container logs, including timestamps.
• Support for forwarding Prometheus metrics.
• Include Memory and CPU limits for container lists.
• Visual updates for the panels, highlighting high CPU and Memory usages
• Filter cgroup stats, forward only container and host metrics.
• Support for multiple Splunk HTTP Event Collector endpoints (support fail-over and load-balancing).
• Handle HTTP Event Collector errors with the incorrect index. Multiple options to redirect to default index, drop or wait.
• Add retry logic to license client to reduce amount of false positive warnings.
• Add HTTP read timeouts (handle gateway timeouts, 504).
• Performance optimizations.
• Optimize payloads for higher write throughput.
• Fixed: fail to parse the latest line in the JSON log.
• Better error handling incorrect configurations.
• Deprecating Join rules in favour of annotations.
• Support for HTTP Event Collector client certificates.
• Fixed: limit directory walkers for depth (fixing issues when directory has a mount to itself)
• Fixed: add a limit of the maximum line size that collector can read at once (defaults to 1Mb).
• Fixed: acknowledgement database stores now NodeID, DevID and a parent folder identifier. That way if NodeID is going to be reused right away - we will identify this file as a new one, if it is in different location.
• Change: docker_stream field has been renamed to stream for compatibility with other container runtime.

3.0 - 2018-02-07

Requires collectorfordocker version 3.0 or above

• Added CPU Quota, CPU Shares, Throttled and Memory Limit Overlays on Container Dashboards.
• Performance improvement on Dashboards by combining multiple charts using one common search.
• New "Review/Privileged containers and enabled capabilities" dashboard to list all privileged containers and enabled security capabilities for containers.
• Fixed bug on Process Dashboard, some charts did not filter by host.
• "Overview: Process" dashboard did not use Span token for timechart dashboards.
• "Top: Containers" fixed incorrect memory usage (showed double size)
• Added alerts in application for notification about outdated collector versions and expired licenses for collector.
• Hide Wait Read/Write IO panels, when this data is not available.
• In process Dashboard show VmRSS with RssAnon, RssFile, and RssShmem.

Collector updates:

• Support for Splunk indexing acknowledgment.
• HTTP Proxy support for License server and Splunk output.
• Allow to configure destination indices for different types of data in collector configuration (stats, logs, host logs, proc stats and events).
• Handling responses from HTTP Event Collector to skip invalid events (will be logged).
• If container is running, but Docker does not provide metadata, allow to wait for metadata.
• Collect security capabilities and uid/gid.
• Support for custom labels, specified with collector configuration.
• Support for partial logs without join rules.
• Bug. Use local timezone by default for local syslog files.
• Bug. Fix small memory leak on deleted containers.
• Bug. When collector is failing to send data to Splunk, impossible to stop collector with terminate.

collectorfordocker 3.0.94.180730

• Show the index name in the output, when Splunk reports incorrect index.

collectorfordocker 3.0.93

• Fixed: Support for Docker running on CentOS (metadata is not attached to metrics).

collectorfordocker 3.0.91

• Fixed: Messages "WARN ... proc.go:441: Unparsable line from /rootfs/proc/X/status" caused by new Linux kernel that reports empty line in proc file system.
• Add HTTP read timeouts (handle gateway timeouts, 504).
• Correctly parse HTTP Event Responses when one of few events fail to be indexed (as an example, wrong index).

2.1 - 2017-10-22

Requires collectorfordocker version 2.1.59.171210 or above

• Implemented collectors dashboard to track number of collectors, their versions and used licenses.
• Fallback to the process IO statistics when blkio is not available.
• Fix IO statistic graphs, showed average, when sum should be used.
• collector - Improved resistance for storage failures.
• collector - License checks reporting.

2.0 - 2017-10-22

Requires collectorfordocker version 2.0.37.171023 or above

• Better labels support in Dashboards. Collector has a breaking feature, replacing format for labels from docker_labels_LABEL1=VALUE1 to docker_labels=[LABEL1=VALUE1,LABEL2=VALUE2].
• Process level metrics.
• Uptime for hosts and processes.
• Fields extraction and support in dashboards for docker daemon (setup host logs collection with collector).
• New top dashboards allow to monitor Hosts/Containers/Processes in real-time.
• Improved dashboards navigation.
• Support for host logs.
• Other bugs and improvements based on user feedback.

Links

• Documentation
• FAQ and the common questions
• License agreement
• Pricing
• Contact