Outcold Solutions LLC

Monitoring Kubernetes - Version 4

Troubleshooting Collector for Kubernetes

Pod is not getting scheduled

Verify that daemonsets have scheduled pods on the nodes

oc get daemonset --namespace collectorforkubernetes

If in the output numbers under DESIRED, CURRENT, READY or UP-TO-DATE are 0, something can be wrong with configuration

NAME                            DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
collectorforkubernetes          0         0         0         0            0           <none>          1m
collectorforkubernetes-master   0         0         0         0            0           <none>          1m

You can run command to describe current state of the daemonset/collectorforkubernetes

$ oc describe daemonsets --namespace collectorforkubernetes

In the output there are will be two daemonsets. In each you can find in the last lines events reported for this daemonset, for example

...
Events:
  Type     Reason            Age                From                  Message
  ----     ------            ----               ----                  -------
  Warning  FailedCreate      31m                daemonset-controller  Error creating: pods "collectorforkubernetes-" is forbidden: SecurityContext.RunAsUser is forbidden

This error means that you are using Pod Security Policies, in that case you need to add our Cluster Role to the privileged Pod Security Policy, with

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: collectorforkubernetes
  name: collectorforkubernetes
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - privileged
- apiGroups:
  ...

Failed to pull the image

When you run command

$ oc get daemonsets --namespace collectorforkubernetes

You can find that number under READY does not match DESIRED

NAMESPACE   NAME                     DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
default     collectorforkubernetes   1         1         0         1            0           <none>          6m

Try to find the pods, that Kubernetes failed to start

$ oc get pods --namespace collectorforkubernetes

If you see that collectorforkubernetes- pod has an error ImagePullBackOff, as in the example below

NAMESPACE   NAME                             READY     STATUS             RESTARTS   AGE
default     collectorforkubernetes-55t61     0/1       ImagePullBackOff   0          2m

In that case you need to verify that your Kubernetes cluster have access to the hub.docker.com registry.

You can run command

$ oc describe pods --namespace collectorforkubernetes

Which should show you an output for each pod, including events raised for every pod

Events:
  FirstSeen LastSeen    Count   From            SubObjectPath               Type        Reason      Message
  --------- --------    -----   ----            -------------               --------    ------      -------
  3m        2m      4   kubelet, localhost  spec.containers{collectorforkubernetes} Normal      Pulling     pulling image "hub.docker.com/outcoldsolutions/collectorforkubernetes:4.0.174.180730"
  3m        1m      6   kubelet, localhost  spec.containers{collectorforkubernetes} Normal      BackOff     Back-off pulling image "rhub.docker.com/outcoldsolutions/collectorforkubernetes:4.0.174.180730"
  3m        1m      11  kubelet, localhost                      Warning     FailedSync  Error syncing pod

Blocked access to external registries

If you are blocking external registries (hub.docker.com) for security reasons, you can copy image from external registry to your own repository with one host which have access to external registry

Copying image from hub.docker.com to your own registry

$ docker pull outcoldsolutions/collectorforkubernetes:4.0.174.180730

After that you can re-tag it by prefixing with your own registry

docker tag  outcoldsolutions/collectorforkubernetes:4.0.174.180730 [YOUR_REGISTRY]/outcoldsolutions/collectorforkubernetes:4.0.174.180730

And push it to your registry

docker push [YOUR_REGISTRY]/outcoldsolutions/collectorforkubernetes:4.0.174.180730

After that you will need to change your configuration yaml file to specify that you want to use image from different location

image: [YOUR_REGISTRY]/outcoldsolutions/collectorforkubernetes:4.0.174.180730

If you need to move image between computers you can export it to tar file

$ docker image save outcoldsolutions/collectorforkubernetes:4.0.174.180730 > collectorforkubernetes.tar

And load it on different docker host

$ cat collectorforkubernetes.tar | docker image load

Pod is crashing or running, but you don't see any data

Start from looking on the logs of collector, this is how the normal output looks like

$ oc logs -f collectorforkubernetes-gvhgw --namespace collectorforkubernetes
INFO 2018/01/24 02:40:17.547485 main.go:213: Build date = 180116, version = 2.1.65


You are running trial version of this software.
Trial version valid for 30 days.

Contact sales@outcoldsolutions.com to purchase the license or extend trial.

See details on https://www.outcoldsolutions.com

INFO 2018/01/24 02:40:17.553805 main.go:207: InstanceID = 2K69F0F36DFT7E1RDBL9MSNROC, created = 2018-01-24 00:29:18.635604451 +0000 UTC
INFO 2018/01/24 02:40:17.681765 watcher.go:95: watching /rootfs/var/lib/docker/containers//(glob = */*-json.log*, match = )
INFO 2018/01/24 02:40:17.681798 watcher.go:95: watching /rootfs/var/log//(glob = , match = ^(syslog|messages)(.\d+)?$)
INFO 2018/01/24 02:40:17.681803 watcher.go:95: watching /rootfs/var/log//(glob = , match = ^[\w]+\.log(.\d+)?$)
INFO 2018/01/24 02:40:17.682663 watcher.go:150: added file /rootfs/var/lib/docker/containers/054e899d52626c2806400ec10f53df29dfa002ca28d08765facf404848967069/054e899d52626c2806400ec10f53df29dfa002ca28d08765facf404848967069-json.log
INFO 2018/01/24 02:40:17.682854 watcher.go:150: added file /rootfs/var/lib/docker/containers/0acb2dc45e1a180379f4e8c4604f4c73d76572957bce4a36cef65eadc927813d/0acb2dc45e1a180379f4e8c4604f4c73d76572957bce4a36cef65eadc927813d-json.log
INFO 2018/01/24 02:40:17.683300 watcher.go:150: added file /rootfs/var/log/userdata.log
INFO 2018/01/24 02:40:17.683357 watcher.go:150: added file /rootfs/var/log/yum.log
INFO 2018/01/24 02:40:17.683406 watcher.go:150: added file /rootfs/var/lib/docker/containers/14fe43366ab9305ecd486146ab2464377c59fe20592091739d8f51a323d2fb18/14fe43366ab9305ecd486146ab2464377c59fe20592091739d8f51a323d2fb18-json.log
INFO 2018/01/24 02:40:17.683860 watcher.go:150: added file /rootfs/var/lib/docker/containers/3ea123d8b5b21d04b6a2b6089a681744cd9d2829229e9f586b3ed1ac96b3ec02/3ea123d8b5b21d04b6a2b6089a681744cd9d2829229e9f586b3ed1ac96b3ec02-json.log
INFO 2018/01/24 02:40:17.683994 watcher.go:150: added file /rootfs/var/lib/docker/containers/4d6c5b7728ea14423f2039361da3c242362acceea7dd4a3209333a9f47d62f4f/4d6c5b7728ea14423f2039361da3c242362acceea7dd4a3209333a9f47d62f4f-json.log
INFO 2018/01/24 02:40:17.684166 watcher.go:150: added file /rootfs/var/lib/docker/containers/5781cb8252f2fe5bdd71d62415a7e2339a102f51c196701314e62a1cd6a5dd3f/5781cb8252f2fe5bdd71d62415a7e2339a102f51c196701314e62a1cd6a5dd3f-json.log
INFO 2018/01/24 02:40:17.685787 watcher.go:150: added file /rootfs/var/lib/docker/containers/6e3eacd5c86a33261e1d5ce76152d81c33cc08ec33ab316a2a27fff8e69a5b77/6e3eacd5c86a33261e1d5ce76152d81c33cc08ec33ab316a2a27fff8e69a5b77-json.log
INFO 2018/01/24 02:40:17.686062 watcher.go:150: added file /rootfs/var/lib/docker/containers/7151d7ce1342d84ceb8e563cbb164732e23d79baf71fce36d42d8de70b86da0f/7151d7ce1342d84ceb8e563cbb164732e23d79baf71fce36d42d8de70b86da0f-json.log
INFO 2018/01/24 02:40:17.687023 watcher.go:150: added file /rootfs/var/lib/docker/containers/d65e4efb5b3d84705daf342ae1a3640f6872e9195b770498a47e2a2d10b925e3/d65e4efb5b3d84705daf342ae1a3640f6872e9195b770498a47e2a2d10b925e3-json.log
INFO 2018/01/24 02:40:17.944910 license_check_pipe.go:102: license-check kubernetes  1 1519345758 2K69F0F36DFT7E1RDBL9MSNROC 1516753758 1516761617 2.1.65 1516060800 true true 0 

In case if you will forget to set url and token for Splunk output, you will see

INFO 2018/01/24 05:08:14.254306 main.go:213: Build date = 180116, version = 2.1.65
Configuration validation failed
[output.splunk]/url is required

In case if connection is failed to our license server you will see that in the logs. If your containers and hosts do not have access to the internet, please contact us for a license which does not require internet access.

If connection will fail to your Splunk instances, you will see that too in logs.

If you don't see mentioning of any *-json.log files, but you have containers running, possible you have journald logging driver enabled instead of json-file. As an example

INFO 2018/01/25 02:51:21.749190 main.go:213: Build date = 180116, version = 2.1.65
You are running trial version of this software.
Trial version valid for 30 days.
Contact sales@outcoldsolutions.com to purchase the license or extend trial.
See details on https://www.outcoldsolutions.com
INFO 2018/01/25 02:51:21.756258 main.go:207: InstanceID = 2K6ERLN622EBISIITVQE34PHA4, created = 2018-01-25 02:51:21.755847967 +0000 UTC m=+0.010852259
INFO 2018/01/25 02:51:21.910598 watcher.go:95: watching /rootfs/var/lib/docker/containers//(glob = */*-json.log*, match = )
INFO 2018/01/25 02:51:21.910909 watcher.go:95: watching /rootfs/var/log//(glob = , match = ^(syslog|messages)(.\d+)?$)
INFO 2018/01/25 02:51:21.910915 watcher.go:95: watching /rootfs/var/log//(glob = , match = ^[\w]+\.log(.\d+)?$)
INFO 2018/01/25 02:51:21.914101 watcher.go:150: added file /rootfs/var/log/userdata.log
INFO 2018/01/25 02:51:21.914354 watcher.go:150: added file /rootfs/var/log/yum.log
INFO 2018/01/25 02:51:22.468489 license_check_pipe.go:102: license-check kubernetes  1 1519440681 2K6ERLN622EBISIITVQE34PHA4 1516848681 1516848681 2.1.65 1516060800 true true 0 

If you don't see any errors, but you don't see any data in the Monitoring Kubernetes application, it is possible that you have specified different index than main for the Splunk HTTP Event Collector token you use. In that case you can add this index as a default index for the Splunk role you are using. Or change our macros in the application to prefix them with index=your_index, you can find the macros in Splunk Web UI, under Setting, Advanced Search, Search Macros. As an example for macro macro_kubernetes_logs you will need to change the value from (sourcetype=kubernetes_logs) to (index=your_index sourcetype=kubernetes_logs). All our dashboards are built on top of these macros, changing that should have immediate effect on the application.


About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which gives you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which helps developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.