Outcold Solutions Blog
Release notes, implementation guides, and practical monitoring notes for Kubernetes, OpenShift, Docker, and Splunk.
Release notes, implementation details, and hard-won operational notes.
Collectord 26.10 ships today for Monitoring Kubernetes, OpenShift, Docker and Linux, for the new Monitoring macOS, and for the ElasticSearch, OpenSearch and syslog forwarders. It is the first release that brings together work we started in 2019, work that kept waiting on other work.
Your Macs might be a build farm, CI runners, an inference cluster, virtual machine hosts, or the laptops IT keeps healthy. Monitoring macOS brings them into Splunk. Collectord on each Mac forwards what the chip draws and how hot it runs, what each process and virtual machine uses, the unified log, crash reports and logins, and the Splunk app turns that into dashboards and alerts. It ships today with Collectord 26.10.1, and the documentation is here.
Most of what Collectord sends to Splunk is logs, and Splunk licenses the text of each event. When a workload writes the same line over and over - a health check, a “waiting for leader” loop, a reconnect message - every copy costs license and disk. Collectord 26.10 adds three ways to find that noise and send less of it, each off until you switch it on.
Each also costs something: CPU and memory in Collectord, and the report has license of its own. Switched on everywhere, they can cost more than they save. Here is what we measured, so you can switch each on where it pays.
Your security team works in Splunk Enterprise Security, and Enterprise Security reads data models. Collectord already forwards the SSH logins on your nodes, your clusters’ API audit, your hosts’ stats and your Macs’ events, but none of it was mapped to the Common Information Model (CIM), so Enterprise Security could not see it. In Collectord 26.10, the Monitoring Kubernetes, OpenShift, Docker, Linux and macOS apps map that data with eventtypes, tags and search-time fields. Nothing changes at collection, so the mapping adds nothing to your Splunk license; the only cost is the data model acceleration you turn on.
Hello, dear reader! I’m Scott, a fairly recent addition to the Outcold Solutions team. Shortly after I joined in March, we decided to attend .conf26 in Denver as a sponsor, and show off our products. We’re a small company, and .conf is a big, busy event. That raised an important question: How do you show products that are supposed to be invisible?
Collectord just quietly does its job.
If we made ceiling fans or warning lights, we could easily show our products at a flashy attention-grabbing booth, but our products aren’t flashy. They’re functional. The whole idea is that they don’t need your attention.
This is going to take some work… and a soldering iron.
Getting AWS data into Splunk usually means ingesting it: the add-on polls the APIs, Firehose streams the logs, and every byte lands in an index and counts against your license. That is the right answer for the data you keep. But a lot of questions only need a live answer. Which instances are running in eu-west-1 right now? Which Lambda functions started timing out after this morning’s deploy? Who signed in as root yesterday? What did EC2 cost us last month?
Search for AWS answers those from the Splunk search bar. Its commands call the AWS APIs as you search and stream the answer into SPL, next to everything Splunk already holds, with nothing written to an index. We showed it at .conf26 and ran it as a beta for the month since. Version 1.0 is on Splunkbase now, and the documentation is here.
Splunk 10.6 released on September 30th, and brought with it a rather abrupt change: the removal of the Analytics Workspace.
That’s okay. Outcold Solutions has a solution.
Introducing… Metrics Explorer
Most of the work in Splunk is translation. A question becomes SPL. A new sourcetype becomes a dashboard. An alert that fired becomes an investigation: which host, since when, is it still happening. Language models are good at that translation. The open questions are which model, where your data goes, and what the assistant is allowed to change.
OS AI Agent is our answer. It runs inside Splunk, on the model you choose, with each user’s own Splunk permissions, and in Chat it changes nothing until you approve. We showed the beta at .conf26; version 1.0 is on Splunkbase now, and the documentation is here.
Splunk 10.6 replaces the MongoDB behind the KV Store with PostgreSQL. The REST API stays the same, and most apps keep working. We tested one of our apps, which keeps all of its state in the KV Store, on Splunk 9.4.15, 10.4.3 and 10.6.0.5, on a single instance and on a search head cluster. These are the five things we have learned so far that admins and app developers should know about.
Last week we were in Denver for Splunk .conf26, September 14 to 17 at the Colorado Convention Center, as a sponsor. We had a booth on the expo floor, Scott gave a session on building dashboards with AI, and we sponsored the happy hour on Tuesday. Thank you to everyone who stopped by: the questions were sharp, the conversations were long, and a good number of you had been running Collectord for years and came to see what is next.
This post is the recap for those who asked for one, and for those who could not make it.
Outcold Solutions builds applications for Splunk Enterprise and Splunk Cloud. Our certified monitoring solutions, powered by Collectord, bring logs, metrics and events from Kubernetes, OpenShift and Docker clusters, Linux and macOS hosts, and Windows containers into Splunk, with the dashboards and alerts that help developers watch their applications and operators keep their clusters healthy. Our search apps query Kubernetes and AWS live from the search bar, with nothing to ingest, and OS AI Agent puts the model you choose to work inside Splunk, with each user's own permissions. Since 2017 we have been helping businesses keep what they need to answer complex questions about their infrastructure in one place.
