Monitoring Docker - Release History
- Monitoring Docker - Release History
- 5.16.353 - 2021-02-11
- 5.16.351 - 2021-01-04
- 5.16.350 - 2020-12-14
- 5.15.305 - 2021-01-04
- 5.15.304 - 2020-09-30
- 5.15.303 - 2020-08-12
- 5.15.301 - 2020-06-24
- 5.15.300 - 2020-06-01
- 5.14.285 - 2020-08-12
- 5.14.284 - 2020-03-23
- 5.14.280 - 2020-01-27
- 5.12.273 - 2019-12-16
- 5.12.272 - 2019-11-08
- 5.12.271 - 2019-11-07
- 5.12.270 - 2019-10-22
- 5.11.266 - 2020-10-15
- 5.11.265 - 2020-06-24
- 5.11.264 - 2019-11-08
- 5.11.261 - 2019-09-13
- 5.11.260 - 2019-09-09
- 5.10.255 - 2019-11-20
- 5.10.253 - 2019-07-31
- 5.10.252 - 2019-07-24
- 5.10.251 - 2019-06-20
- 5.10.250 - 2019-06-18
- 5.9.240 - 2019-05-14
- 5.8.231 - 2019-04-25
- 5.8.230 - 2019-04-22
- 5.7.220 - 2019-03-18
- 5.6.213 - 2019-03-03
- 5.6.212 - 2019-02-19
- 5.5.205 - 2019-01-25
- 5.5.203 - 2019-01-25
- 5.5.202 - 2019-01-24
- 5.4.201 - 2018-12-19
- 5.4 - 2018-12-17
- 5.3 - 2018-11-19
- 5.2 - 2018-10-15
- 5.1 - 2018-09-17
- 5.0 - 2018-09-03
- 3.0 - 2018-02-07
- 2.1 - 2017-10-22
- 2.0 - 2017-10-22
- Links
5.16.353 - 2021-02-11
Collectord updates:
- Bug fix: collectord can report parse int errors on the stderr
- Upgrade go runtime to 1.15.8
5.16.351 - 2021-01-04
Collectord updates:
- Bug fix: host file inputs can raise a fatal error: concurrent map writes
5.16.350 - 2020-12-14
Requires collectorfordocker version 5.16.350 or above
- New dashboard: Collectord metrics
Collectord updates:
- Annotations for collecting prometheus metrics: authorization keys and CAName for SSL certificates
- Improvement for DNS resolutions of Splunk output FQDN
- Export internal collectord metrics in Prometheus format
- Forwarding internal collectord metrics to Splunk
- For the watch objects inputs being able to hide management fields
- In the diag include all open file descriptors
- Upgrade go runtime to 1.14.13
- Remove
\0
symbol from the labels values in the prometheus metrics - Allow to filter host logs with blacklist and whitelist
- Bug fix: less verbose warnings about not being able to load resources from API server
- Bug fix: performance improvements for Ack DB
- Bug fix: custom prometheus metrics forwarded by Collectord do not include cluster field or custom user fields
- Bug fix: addon pod terminates faster
- Bug fix: verify command trying to post to all outputs with all indexes specified in the configuration
- Bug fix: crash in AckDB
- Bug fix: input system stats does not recognize ouputs specified for the host and cgroup
- Bug fix: verify command runs recursively all the time for host logs even when recursively is set to false
5.15.305 - 2021-01-04
Collectord updates:
- Upgrade go runtime to 1.14.13
- Bug fix: host file inputs can raise a fatal error: concurrent map writes
5.15.304 - 2020-09-30
Collectord updates:
- Bug fix: cannot disable forwarding of the api resources
5.15.303 - 2020-08-12
Collectord updates:
- Upgrade golang to 1.14.7 to fix the hang in runtime
5.15.301 - 2020-06-24
Collectord updates:
- Bug fix: verify command broken for addon pod
5.15.300 - 2020-06-01
Requires collectorforkubernetes version 5.15.300 or above
Collectord updates:
- Support for annotations to add custom user fields to data
- Support for blacklisting and whitelisting Prometheus metrics (significally reducing the indexing cost of data)
- Verify command improvements - verify proper configurations for cgroup (memory/memory.use_hierarchy is 1)
- Bug fix: fix bug in prometheus metrics parser, empty fields can be filled with previous fields
- Bug fix: occasionally addon can report warnings about trying to delete expired keys from ack db
- Bug fix: better handle of connections to metrics endpoints exported in Prometheus format
- Bug fix: http connections improvements for when Splunk is unresponsive
- Bug fix: broken diag
5.14.285 - 2020-08-12
Collectord updates:
- Upgrade golang to 1.14.7 to fix the hang in runtime
5.14.284 - 2020-03-23
Collectord updates:
- New annotation to configure whitelist pattern for log messages
- Allow to override Kubernetes service URL
- Bug fix: panic in output for addon
- Bug fix: performance and memory usage improvement for ack db
5.14.280 - 2020-01-27
Requires collectorforkubernetes version 5.14.280 or above
- Logs dashboard: filters depend on selection
Collectord updates:
- Support templates in the index, source and sourcetype
- Allow to exclude indexed fields when forwarding to Splunk
- Support annotation for stats interval for containers
- Bug fix: verify command can show incorrect error about verifying journald input
- Bug fix: index on namespace should set index for application logs
5.12.273 - 2019-12-16
Requires collectorfordocker version 5.12.272 or above
- Bug fix: Swarm Services dashboard compatibility with the new format of metrics
5.12.272 - 2019-11-08
Collectord updates:
- Bug fix: in case when the rotated files are reusing FileID/DevID Collectord stops forwarding rotated files
5.12.271 - 2019-11-07
Requires collectorfordocker version 5.12.271 or above
- Improvements for the macros for backward compatibility
- Bug fix: Swarm Services dashboard could not filter based on the stack name
Collectord updates:
- Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
- Bug fix: reduce warnings failed to get the new event in pipeline - submitted
- Stability improvements
5.12.270 - 2019-10-22
Requires collectorfordocker version 5.12.27- or above
- Compact metrics (pre-calculated on Collectord side)
- Switched stats for host and cgroup in different macros
- Containers count for hosts on Hosts and Host dashboards
- Improve performance of the search in Docker Services and ECS Services dashboards
- Use base macro for alerts
- Add cluster name in the alert results
Collectord updates:
- Describe command to see applied annotations for containers
- Bug fix: panic when pipe join configuration is removed
- Bug fix: panic when proc stats is enabled and cgroup stats is disabled
- Bug fix: support ProxyBasicAuthorization for license server checks
- Bug fix: Fix for collecting first sample (can show high CPU usage for first sample)
- Beta: dynamic index, source and sourcetype names based on the metafields
- Beta: cluster diagnostics with one rule: node entropy
5.11.266 - 2020-10-15
Collectord updates:
- Upgrade golang to 1.14.10 to fix the hang in runtime
5.11.265 - 2020-06-24
Collectord updates:
- Bug fix: memory improvement for large ackdb files
5.11.264 - 2019-11-08
Collectord updates:
- Bug fix: in case when the rotated files are reusing FileID/DevID Collectord stops forwarding rotated files
5.11.261 - 2019-09-13
Collectord update:
- Bug fix: improves logging for the directory walker
5.11.260 - 2019-09-09
Requires collectorfordocker version 5.11.260 or above
- Bug fix: changing source type does not allow to use Swarm and ECS services dashboards
- Bug fix: improving usability of Swarm and ECS services dashboards for large deployments
Collectord updates:
- Bug fix: duplicate events then pipeline is getting throttled
- Bug fix: don't use throttling for devnull output
- Bug fix: better recovery for ack db corruption
- Bug fix: crash on journald input initialization when ack db is corrupted
- Bug fix: annotations joinmultiline requires joinpartial
- Bug fix: configurations for stdout only with annotations can crash collectord
- Set events = 50 by default for Splunk output batches
5.10.255 - 2019-11-20
Collectord updates:
- Bug fix: better recovery for ack db corruption
- Bug fix: crash on journald input initialization when ack db is corrupted
5.10.253 - 2019-07-31
Collectord update:
- Bug fix: collectord can pick up compressed json logs (*.gz)
- Bug fix: too verbose warnings from the docker watcher about retries
5.10.252 - 2019-07-24
Collectord update:
- Support for configuring the thruput (general and with annotations for container logs)
- Support for configuring too old or too new events (general and with annotations for container logs)
5.10.251 - 2019-06-20
Collectord update:
- Ability to configure Acknowledgement database for collectord.
5.10.250 - 2019-06-18
Requires collectorforopenshift version 5.10.250 or above
- Cluster field filters
- Base macro for overriding macros for other macros
Collectord updates:
- Support for volatile and persistent journald storage with default configuration
- Updated YAML configuration to include most common resources
- Better support for overriding sourcetype, that does not require to update the Splunk macros
- Bug fix: rarely when collectord fails to post to HEC it can panic
- Bug fix: better support for OpenShift 4.x and CRI-O storage
- Bug fix: space characters in index annotations can break the pipeline
5.9.240 - 2019-05-14
Requires collectorfordocker version 5.9.240 or above
- Visual improvements on the graphs for the number of logs and events
Collectord updates:
- Support for multiple Splunk destinations (outputs)
- Support subdomains for annotations (to deploy multiple collectord instances)
- Bug fix: journald input keeps fd open to the rotated files
- Bug fix: fix in the annotation parser for the interval annotations
- Bug fix: fix splunk url selection configuration for multiple splunk URLs
5.8.231 - 2019-04-25
- Bug fix: Collectord usage report shows trial licenses for all instances
5.8.230 - 2019-04-22
Requires collectorfordocker version 5.8.230 or above
- Bug fix: Swarm dashboard does not render containers, when namespace field is not available.
- Use multiselect filters for most dashboards and filters with possibility to input custom filters.
- Reduce dedup usage to improve performance on dashboards.
Collectord updates:
- Bug fix: clogging collectord output with errors when incorrect index is used.
- Bug fix: short lived containers can results in duplicating logs.
- Bug fix: clogging collectord output with warnings when kernel reports incorrect VmRss size.
- Bug fix: annotations cannot override timestamp location for fields extraction.
- Bug fix: verify command reports Journald input in incorrect place.
- Better support for cgroup symlinks, automatically discover correct location.
5.7.220 - 2019-03-18
Requires collectorfordocker version 5.7.220 or above
- Review savedsearches/alerts to support indexing delay (start searches from 2 minutes behind) and run them in more random time.
- Fixed single value memory panel on host dashboard (missed span)
- Use SEGMENTATION=none for stats events to use less disk space (needs to me moved to indexers)
Collectord updates:
- Support hostname formatting with environment variables in configuration
- New rotated file logic uses less file descriptors and frees rotated files quicker
- Allow to specify a default sampling value for container logs
- Reimplemented shutdown sequence to stop collectord faster
- Allow to override sampling percent with annotations
- New Input: journald
5.6.213 - 2019-03-03
- Collectord: Fix panic, when collectord does not have access to docker socket, and information about this container does not exist on the disk.
5.6.212 - 2019-02-19
Requires collectorfordocker version 5.6.212 or above
- New: Alert: high CPU usage on the host.
- Fixed: Splunk usage dashboard - charts do not show the data, when the used indexed aren't searchable by default.
- New: Support Dark theme.
- New: Free text search in Logs dashboard.
- New: Add auto-refresh options to the dashboard.
- Fixed: Revisited CPU limits and requests for Pods and Containers.
Collectord updates:
- Fixed: auto-recovery from the corrupted write-ahead-log in acknowledgment database.
- New: support sampling (random and hash-based) for container/application and host logs.
- New: when running multiple collectord on one host (with different output) - count that as one licensed host, change InstanceID format.
- Fixed: when container is scheduled with remove flag lock the file till collectord processes it completely.
- Fixed: collectord reports rare warning about unparsable uint64 max value from proc filesystem.
- Fixed: collectord reports rare warning about unparsable line from proc/io files.
5.5.205 - 2019-01-25
- Collectord fix: collectord could stop sending container file logs when the original file has been truncated (using the same Node ID as previously used log file).
5.5.203 - 2019-01-25
- Collectord fix: collectord could send an empty
X-Splunk-Request-Channel
header to Splunk.
5.5.202 - 2019-01-24
Requires collectorfordocker version 5.5.202 or above
- New: Dashboard Services -> AWS ECS. Review containers running as an ECS Service.
- New: Dashboard Services -> Swarm Services. Review containers running as a Swarm Service.
Collectord updates:
- Fixed: Interval 0 in prometheus input can crash the collectord.
- Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from finding the files in the volume.
5.4.201 - 2018-12-19
Requires collectorfordocker version 5.4.201 or above
- Fixed: Alerts for licenses issued with AWS Subscriptions
Collectord updates:
- Fixed: Better handling rotated files (less open fd)
- Fixed: Events input can hang in the err loop.
5.4 - 2018-12-17
Requires collectorfordocker version 5.4 or above
- Compatibility update for collectord 5.4.
Collectord updates:
- New: Attach EC2 metadata fields
- New: Basic Auth for Proxy (License Server and Splunk)
- Fixed: Collectord verify reports CRI-O as unsupported runtime.
- Fixed: Rare crash on Prometheus metrics definition.
- Fixed: Better handling of acknowledgment database corruption.
- Fixed: When handling incorrect indexes, collectord can send index with empty string, that Splunk recognize as incorrect index
5.3 - 2018-11-19
Requires collectorfordocker version 5.3 or above
- New: Alert for showing when Collectord reports errors in Processing pipelines (as an example if it failed to extract fields).
- New: Alert for showing when Collectord reports warnings.
- New: Alert if lag in the indexing of the data.
- New: Splunk Usage (License usage, number of events) report under Setup.
- Fixed: lookup with alerts causing very often replication activities on SHC
Collectord updates:
- Fixed: high memory usage with Gzip compression enabled (reduced memory usage).
- New: Allow to disable pipe.join with annotations.
- Fixed: In high amount of logs (10,000 events per second) Collectord can read lines not in full, that can break JSON logs.
- Fixed: When collectord writes a Warning that it failed to post to Splunk, it will write a Success message after retry.
- New: Allow to hash sensitive data with annotations.
- Fixed: Group network socket tables to reduce the amount of forwarded data (4 times reducing the amount of data)
- Fixed: Identify when glob and match pattern require recursive directory traversal.
- New: Annotation for complete disabling of the handling and forwarding logs for containers.
- Fixed: Collectord showed few Debug messages on start.
- Fixed: Performance improvements for log forwarding (up to 35% in high amount of logs).
- Fixed: reduce duplication of the Kubernetes events, forwarded to Splunk.
- Fixed: Support ECS cgroup matching with the default configuration.
- Fixed: Support docker daemon logs forwarding with the default configuration.
- Fixed: Failed to parse proc name from the stat file with the not paired parentheses.
5.2 - 2018-10-15
Requires collectorfordocker version 5.2 or above
- New: Review/Storage dashboard based on storage metrics.
- New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
- Performance improvements
Collector updates:
- New: runtime storage metrics (usage, available, inodes)
- New: image is built on top of
SCRATCH
image. - New:
verify
anddiag
commands for troubleshooting. - New: support
/dev/null
output for logs - New: override source/sourcetype and index base on regexp pattern for container logs.
- Fixed: do not send empty docker_labels
- New: support docker JSON tags and labels
- Fixed: allowing a new license to unblock collector with the expired license.
- Fixed: Prometheus parser fails to parse metrics with labels that end with a comma.
- Fixed: Performance improvements
- New: Prometheus parser supports basic authentication
- Fixed: Workaround for a bug in HTTP Event Collector, that can return an incorrect index of a failed event
5.1 - 2018-09-17
Requires collectorfordocker version 5.1 or above
- New: Network metrics (MB, Packets, Drops and Errors) for host and containers.
- New: Network socket tables (list of port that containers and hosts are listen on, connections to external resources).
- New: Network review dashboard to see the list of connection to public services and in private network.
- Improvement: Replace python-based lookup with macro written with eval.
- Improvement: Visual improvement for showing when the object was Last Seen (highlighting and showing minutes ago).
- Improvement: Changed source of proc stats to proc root filesystem, to keep minimum list of unique sources.
- New: Support for Splunk multi-threads outputs (for forwarding more than 3000 events per second).
- Improvement: Performance improvements for Prometheus parsing.
- Improvement: Calculate checksums for first bytes of files, to better identify new files with reused iNode.
- Improvement: Reduce amount of metrics forwarded with proc_stats by excluding system threads.
- Improvement: Configuration for gzip compression.
5.0 - 2018-09-03
Requires collectorfordocker version 5.0 or above
- Auto-discover and forward Application logs from host mounts or local volumes.
- Annotations for containers to change per container configurations (index, source, join rules, replaces and more).
- Escaping terminal sequences from container logs.
- Redirecting logs to /dev/null for specific patterns.
- Replace patterns in container and application logs (hiding sensitive or not important information).
- Support for extracting fields from the container logs, including timestamps.
- Support for forwarding Prometheus metrics.
- Include Memory and CPU limits for container lists.
- Visual updates for the panels, highlighting high CPU and Memory usages
- Filter cgroup stats, forward only container and host metrics.
- Support for multiple Splunk HTTP Event Collector endpoints (support fail-over and load-balancing).
- Handle HTTP Event Collector errors with the incorrect index. Multiple options to redirect to default index, drop or wait.
- Add retry logic to license client to reduce amount of false positive warnings.
- Add HTTP read timeouts (handle gateway timeouts, 504).
- Performance optimizations.
- Optimize payloads for higher write throughput.
- Fixed: fail to parse the latest line in the JSON log.
- Better error handling incorrect configurations.
- Deprecating Join rules in favour of annotations.
- Support for HTTP Event Collector client certificates.
- Fixed: limit directory walkers for depth (fixing issues when directory has a mount to itself)
- Fixed: add a limit of the maximum line size that collector can read at once (defaults to 1Mb).
- Fixed: acknowledgement database stores now NodeID, DevID and a parent folder identifier. That way if NodeID is going to be reused right away - we will identify this file as a new one, if it is in different location.
- Change:
docker_stream
field has been renamed tostream
for compatibility with other container runtime.
3.0 - 2018-02-07
Requires collectorfordocker version 3.0 or above
- Added CPU Quota, CPU Shares, Throttled and Memory Limit Overlays on Container Dashboards.
- Performance improvement on Dashboards by combining multiple charts using one common search.
- New "Review/Privileged containers and enabled capabilities" dashboard to list all privileged containers and enabled security capabilities for containers.
- Fixed bug on Process Dashboard, some charts did not filter by host.
- "Overview: Process" dashboard did not use Span token for timechart dashboards.
- "Top: Containers" fixed incorrect memory usage (showed double size)
- Added alerts in application for notification about outdated collector versions and expired licenses for collector.
- Hide Wait Read/Write IO panels, when this data is not available.
- In process Dashboard show VmRSS with RssAnon, RssFile, and RssShmem.
Collector updates:
- Support for Splunk indexing acknowledgment.
- HTTP Proxy support for License server and Splunk output.
- Allow to configure destination indices for different types of data in collector configuration (stats, logs, host logs, proc stats and events).
- Handling responses from HTTP Event Collector to skip invalid events (will be logged).
- If container is running, but Docker does not provide metadata, allow to wait for metadata.
- Collect security capabilities and uid/gid.
- Support for custom labels, specified with collector configuration.
- Support for partial logs without join rules.
- Bug. Use local timezone by default for local syslog files.
- Bug. Fix small memory leak on deleted containers.
- Bug. When collector is failing to send data to Splunk, impossible to stop collector with terminate.
collectorfordocker 3.0.94.180730
- Show the index name in the output, when Splunk reports incorrect index.
collectorfordocker 3.0.93
- Fixed: Support for Docker running on CentOS (metadata is not attached to metrics).
collectorfordocker 3.0.91
- Fixed: Messages "WARN ... proc.go:441: Unparsable line from /rootfs/proc/X/status" caused by new Linux kernel that reports empty line in proc file system.
- Add HTTP read timeouts (handle gateway timeouts, 504).
- Correctly parse HTTP Event Responses when one of few events fail to be indexed (as an example, wrong index).
2.1 - 2017-10-22
Requires collectorfordocker version 2.1.59.171210 or above
- Implemented collectors dashboard to track number of collectors, their versions and used licenses.
- Fallback to the process IO statistics when blkio is not available.
- Fix IO statistic graphs, showed average, when sum should be used.
- collector - Improved resistance for storage failures.
- collector - License checks reporting.
2.0 - 2017-10-22
Requires collectorfordocker version 2.0.37.171023 or above
- Better labels support in Dashboards.
Collector has a breaking feature, replacing format for labels from
docker_labels_LABEL1=VALUE1
todocker_labels=[LABEL1=VALUE1,LABEL2=VALUE2]
. - Process level metrics.
- Uptime for hosts and processes.
- Fields extraction and support in dashboards for docker daemon (setup host logs collection with collector).
- New top dashboards allow to monitor Hosts/Containers/Processes in real-time.
- Improved dashboards navigation.
- Support for host logs.
- Other bugs and improvements based on user feedback.