Outcold Solutions - Monitoring Kubernetes, OpenShift and Docker in Splunk

FAQ and common questions

How to submit a support request?

If you have a support contract, you can submit a support request by sending an email to support@outcoldsolutions.com. When signing a support contract, you need to assign people from your organization who will be able to submit support requests. Please contact your procurement department if you need to add more people.

When you submit a support request, please include the following information:

  • LicenseID. (Not the License Key). You can find it:
    • In our applications, dashboard Setup->Collectord usage
    • In the output of Collectord logs, similar to INFO 2023/05/24 05:36:48.834659 outcoldsolutions.com/collectord/license/license_check_pipe.go:158: license-check openshift BG5183Q89IE2G 0 0 ...
  • If you are a partner or a contractor for a customer, please include the customer name.
  • Incident description in the topic of email.
  • Incident details in the body of the email.
    • The more details you can include in the email, the faster we will be able to help you, as we will not need to ask you for additional information.
    • In 99% of cases, we ask you to provide diagnostic information from the Collectord instance that is running on the node where you are experiencing the issue. This archive will contain most of the information we require, including the version of Collectord and its basic usage. See below for how to collect diagnostic information (all 4 steps are required).
    • If you cannot provide diagnostic information, please provide as much information as possible to describe your issue, and include the version of Collectord that you are using.

To make sure that we can help you as fast as possible, please follow the steps above and:

  • Our primary support channel is email. We can follow up with you via phone call or web meeting. or a web meeting in the support request. We will contact you if we need to.
  • You can keep your team in CC to keep track of your support request. However, please respect our support contract and keep the number of people in CC to a minimum. We will not be able to respond to emails from people who are not covered by the support contract.
  • Keep one support request per email thread. If you have multiple issues, please submit multiple support requests.

I do not see IO metrics on host dashboards

We use metrics from the blkio-controller. It is not always enabled. As a workaround, you can use the sum from proc metrics, which we also collect for processes.

I received a license. How do I apply it?

Collectord reads a license key from the configuration file.

Collectord for Kubernetes and Collectord for OpenShift

If you are using the collectord for OpenShift or Kubernetes, you can set the license in the ConfigMap that we include in our configuration yaml files. Find the line license =, add your license key as a value. License key should not have any spaces. Starting from version 5.0, you do not need to restart the collectors after setting the license. automatically after a few moments.

If you are using a collectord version below 5.0, you need to restart collectors. When you modify just a ConfigMap, it does not trigger the restart of the Pod.

You can just delete all the running pods in our namespace, and scheduler will create new pods.

In case of OpenShift: oc delete --namespace collectorforopenshift pods --all

In case of Kubernetes: kubectl delete --namespace collectorforkubernetes pods --all

If you are using version 3 of our solution for Monitoring OpenShift and Kubernetes, we deploy our collectord in the default namespace. In that case, you will need to manually delete all pods, and they will be rescheduled with the DaemonSet.

Collectord for Docker

For the Docker collectord, you can modify the collector.yaml configuration file if you already have your own. Or you can set the license key with the environment variable --env "COLLECTOR__LICENSE=general__license=...".

How much data does your application generate?

In the application version 5.3 and above, you can find a Splunk Usage dashboard under Setup.

Because we are using Indexed field extractions in HTTP Event Collector, in our tests, we saw less than 5% of Splunk Licensing cost increase for logs.

You can also run a search in Splunk to get the information on how much licensing usage for every source type.

index=_internal source=*metrics.log  | eval MB=kb/1024 | search group="per_sourcetype_thruput" | timechart span=1h sum(MB) by series

In our tests for Kubernetes nodes with 20-30 containers, we have seen around 200MB per day of data indexed for kubernetes_stats and kubernetes_proc_stats together.

In our demo environment with 2 OpenShift clusters (9 nodes total: 1+3 masters, 1+4 worker nodes) and 62 pods, here is the licensing usage breakdown

Splunk Licensing Usage

Performance

Please read our blog post about the performance of our collectord: Forwarding 10,000 1k events generated by containers from a single host with ease

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all container environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and easy-to-deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and help operators keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.