Between July 28 and August 4, 2026, our license server rejected online license checks from Collectord versions 25.10.3 and older with 400 Bad Request. The root cause was a server-side change; licenses themselves remained valid throughout, and Collectord 26.04 and later were not affected. The issue is fixed, all affected agents recovered automatically, and no customer action is required. Here is the full postmortem.

Incident window: July 28, 2026 02:40 UTC - August 4, 2026 10:53 UTC

Status: Resolved. All affected agents recovered automatically; no customer action is required.

Summary

Affected agents logged License check failed. request failed (status = 400 Bad Request) on every hourly license check and accumulated failed-check points.

Collectord tolerates up to 7 days of consecutive failed license checks before it treats the license as invalid, so most agents rode out the entire incident with no impact beyond warning logs. The earliest-affected agents exhausted that window in the early hours of August 4 and paused data forwarding for up to ~9 hours before the fix restored validation. Forwarding resumed automatically; file-based log collection continues from saved positions, so paused logs were generally delivered late rather than lost.

What happened

On July 28 at 02:40 UTC, as part of hardening the license server against ongoing bot attacks, we deployed stricter verification of incoming license-check requests. The hardened verification required every request to carry an explicit signature-scheme version (SignatureVersion).

That field was introduced in Collectord 26.04.1 together with FIPS-compliant image support, which added a second, FIPS-approved signature scheme. Agents released before 26.04.1 predate the field and have always relied on the server defaulting to the original scheme. After the hardening change, their requests were rejected instead of being verified with the original scheme.

The failure was determined by agent version, not by license: the same license validated fine when checked by a 26.04+ agent and failed when checked by an older one - which is why only some environments noticed.

Timeline (UTC)

2026-07-28 02:40 - Hardened license server deployed; rejections of license checks from pre-26.04 agents begin. Agents continue operating normally within their failed-check grace window.
2026-08-04 ~02:40 - The earliest-affected agents exhaust the 7-day grace window, report the license as invalid, and pause data forwarding. First customer reports follow.
2026-08-04 10:53 - Fix deployed: requests without SignatureVersion are once again verified with the original signature scheme. Rejections stop immediately.
2026-08-04, within the following hour - All affected agents validate successfully on their next hourly check and resume normal operation.

Why it took a week to detect

Our license-server alerting monitored server errors (5xx) and overall traffic volume, and both stayed normal: the rejections surfaced as client errors (4xx), which we did not alert on, and affected agents kept checking in on their regular hourly schedule, so traffic never dipped. On the customer side, the agents’ 7-day grace window absorbed the failures silently - working as designed, but it also meant the first customer-visible impact (and the first reports) came only hours before the fix.

What we changed

The license server again accepts license checks without a SignatureVersion and verifies them with the original signature scheme. Regression tests now pin this legacy wire format, including under FIPS-enforced builds.
We added alerting on elevated 4xx rates at the license endpoint. With it in place, this incident would have been detected within about an hour instead of a week.

Detecting this in your own Splunk

The bundled Splunk apps already ship a saved alert for exactly this condition - “Collector Failed License Checks” (in Monitoring Kubernetes, Monitoring OpenShift, Monitoring Docker, and Monitoring Linux). It runs daily, is enabled by default, and fires when any collectord reports more than 24 consecutive failed license checks (about a day of failures). During this incident it would have flagged affected clusters within two days of onset - five days before agents reached enforcement.

Two things to verify in your environment:

The saved search is still enabled in your installed app (Settings → Searches, reports, and alerts).
An alert action is attached (email, Slack, webhook, etc.). Out of the box the alert only records to Splunk’s Triggered Alerts list, so if nobody reviews that list, attach an action that reaches your team.

The companion alerts “Collector License Expiration (less than 14 days)” and “Collector outdated” are also worth keeping enabled.

Recommendation

Per our License Agreement, Enterprise Support covers the most current release plus one prior release. Older agents continue to operate - this incident was a server-side defect, not an agent defect - but running a supported release ensures the fastest diagnosis and access to current fixes and features, including the FIPS-compliant images introduced in 26.04. If you are running 25.10 or older, we recommend planning an upgrade to the current release.

If your agents logged license warnings during the incident window and anything still looks off, contact support@outcoldsolutions.com and we will help directly.

Denis Gladkikh, Co-Founder