For OpenShift use oc tool and namespace collectorforopenshift-syslog

Verify configuration

Get the list of the pods

1$ kubectl get pods -n collectorforkubernetes-syslog
2NAME                                           READY     STATUS    RESTARTS   AGE
3collectorforkubernetes-syslog-addon-857fccb8b9-t9qgq   1/1       Running   1          1h
4collectorforkubernetes-syslog-master-bwmwr             1/1       Running   0          1h
5collectorforkubernetes-syslog-xbnaa                    1/1       Running   0          1h

Considering that we have 3 different deployment types, the DaemonSet we deploy on Masters (collectorforkubernetes-syslog-master), the DaemonSet we deploy on non-master nodes (collectorforkubernetes-syslog) and one Deployment addon (collectorforkubernetes-syslog-addon) verify one node from each deployment (in example below change the pod names to the pods that are running on your cluster).

1$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-addon-857fccb8b9-t9qgq -- /collectord verify
2$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord verify
3$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-xbnaa -- /collectord verify

OpenShift

For each command you will see an output similar to

 1Version = 5.2.176
 2Build date = 181012
 3Environment = kubernetes
 4
 5
 6  General:
 7  + conf: OK
 8  + db: OK
 9  + db-meta: OK
10  + instanceID: OK
11    instanceID = 2LEKCFD4KT4MUBIAQSUG7GRSAG
12  + license load: OK
13    trial
14  + license expiration: OK
15    license expires 2018-11-12 15:51:18.200772266 -0500 EST
16  + license connection: OK
17
18  Kubernetes configuration:
19  + api: OK
20  + pod cgroup: OK
21    pods = 18
22  + container cgroup: OK
23    containers = 39
24  + volumes root: OK
25  + runtime: OK
26    docker
27
28  Docker configuration:
29  + connect: OK
30    containers = 43
31  + path: OK
32  + cgroup: OK
33    containers = 40
34  + files: OK
35
36  CRI-O configuration:
37  - ignored: OK
38    kubernetes uses other container runtime
39
40  File Inputs:
41  x input(syslog): FAILED
42    no matches
43  + input(logs): OK
44    path /rootfs/var/log/
45  
46Errors: 1

With the number of the errors at the end. In our example we show output from minikube, where we see some invalid configurations, like

input(syslog) - minikube does not persist syslog output to disk, we will not be able to see these logs in applicatione

If you find some error in the configuration, after applying the change kubectl apply -f ./collectorforkubernetes-syslog.yaml you will need to recreate pods, for that you can just delete all of them in our namespace kubectl delete pods --all -n collectorforkubernetes. The workloads will recreate them.

Describe command

When you apply annotations through the namespace, workload, configurations and pods it could be hard to track which annotations are applied to the Pod or Container. You can use a describe command of collectord to get information which annotations are used for the specific Pod. You can use any collectord Pod to run this command on the cluster

1kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres

Collect diagnostic information

If you need to open a support case you can collect diagnostic information, including performance, metrics and configuration (excluding splunk URL and Token).

1. Collect diagnostics information run following command

Choose pod from which you want to collect a diag information.

The following command takes several minutes.

1kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord diag --stream 1>diag.tar.gz

You can extract a tar archive to verify the information that we collect. We include information about performance, memory usage, basic telemetry metrics, information file with the information of the host Linux version and basic information about the license.

2. Collect logs

1kubectl logs -n collectorforkubernetes-syslog --timestamps collectorforkubernetes-syslog-master-bwmwr  1>collectorforkubernetes.log 2>&1

3. Run verify

1kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord verify > verify.log

4. Prepare tar archive

1tar -czvf collectorforkubernetes-$(date +%s).tar.gz verify.log collectorforkubernetes.log diag.tar.gz

Forwarding Kubernetes logs to QRadar (syslog)

Troubleshooting