Kubeye

Kubeye is a tool for inspecting Kubernetes clusters. It runs a variety of checks to ensure that Kubernetes pods are configured using best practices, helping you avoid problems in the future. Quickly get cluster core component status and cluster size information and abnormal Pods information and tons of node problems. Developed by the GO language. Support for user-defined best practice configuration rules and the addition of cluster fault scouts, which can refer to the Node-Problem-Detector project。

Usage

1、Get the Installer Excutable File

• Binary downloads of the kubeye can be found on the Releases page. Unpack the binary and you are good to go!

• Build Binary from Source Code

git clone https://github.com/kubesphere/kubeye.git
cd kubeye 
make ke-linux

2、Perform operation

./ke audit --kubeconfig /home/ubuntu/.kube/config

3、(Optional) Install Node-problem-Detector in the inspection cluster

Note: The NPD module does not need to be installed When more detailed node information does not need to be probed.

./ke install npd --kubeconfig /home/ubuntu/.kube/config

What kubeye can do

1. Core component detection in the cluster, including controller-manager, scheduler and ETCD exception detection.
2. Node detection in the cluster, including Kubelet abnormalities, insufficient machine MEMORY/CPU/DISk resources, docker service exceptions.
3. Pod detection int the cluster, including pod best practices, pod exceptions information.

Features

YES/NO	CHECK ITEM	Description
✅	ETCDHealthStatus	If ETCD is abnormal, displays dial tcp 192.168.13.8:2379: connect: connection refused
✅	Controller-ManagerHealthStatus	If Controller-Manager is abnormal, displays dial tcp 127.0.0.1:10252: connect: connection refused
✅	ScheduleHealthStatus	If Schedule is abnormal, displays dial tcp 127.0.0.1:10251: connect: connection refused
✅	NodeMemory	If the node is full of Memory, the node displays NotReady
✅	DockerHealthStatus	If docker is abnormal, displays cannot connect to the Docker daemon at unix:///var/run/docker.sock
✅	NodeDisk	If the node is full of Disk, displays FreeDiskSpaceFailed
✅	KubeletHealthStatus	If kubelet not work, the node displays NotReady
✅	NodeCPU	If the node CPU is always full, the node displays NotReady
✅	NodeCorruptOverlay2	Overlay2 is not available
✅	NodeKernelNULLPointer	the node displays NotReady
✅	NodeDeadlock	A deadlock is a phenomenon in which two or more processes are waiting for each other as they compete for resources
✅	NodeOOM	Monitor processes that consume too much memory, especially those that consume a lot of memory very quickly, and the kernel kill them to prevent them from running out of memory
✅	NodeExt4Error	Ext4 mount error
✅	NodeTaskHung	Check to see if there is a process in state D for more than 120s
✅	NodeUnregisterNetDevice	Check corresponding net
✅	NodeCorruptDockerImage	Check docker image
✅	NodeAUFSUmountHung	Check storage
✅	NodeDockerHung	Docker hung, you can check docker log
✅	PodSetLiveNessProbe	No livenessProbe was declared
✅	PodSetTagNotSpecified	The mirror address does not declare tag or tag is latest
✅	PodSetRunAsPrivileged	Running a pod in a privileged mode means that the pod can access the host’s resources and kernel capabilities
✅	PodSetImagePullBackOff	Pod can't pull the image properly, so it can be pulled manually on the corresponding node
✅	PodSetImageRegistry	Checks if the image form is at the beginning of the corresponding harbor
✅	PodSetCpuLimitsMissing	No CPU Resource limit was declared
✅	PodNoSuchFileOrDirectory	Go into the container to see if the corresponding file exists
✅	PodIOError	This is usually due to file IO performance bottlenecks
✅	PodNoSuchDeviceOrAddress	Check corresponding net
✅	PodInvalidArgument	Check the storage
✅	PodDeviceOrResourceBusy	Check corresponding dirctory and PID
✅	PodFileExists	Check for existing files
✅	PodTooManyOpenFiles	The number of file /socket connections opened by the program exceeds the system set value
✅	PodNoSpaceLeftOnDevice	Check for disk and inode usage
	NodeTokenExpired	Token certificate expired
	NodeApiServerExpired	kube-apiserver certificate expired
	NodeKubeletExpired	Kubelet certificate expired
	PodSetCpuRequestsMissing	The CPU Resource Request value was not declared
	PodSetHostIPCSet	Set the hostIP
	PodSetHostNetworkSet	Set the hostNetwork
	PodHostPIDSet	Set the hostPID
	PodMemoryRequestsMiss	No memory Resource Request value is declared
	PodSetHostPort	Set the hostPort
	PodSetMemoryLimitsMissing	No memory Resource limit value is declared
	PodNotReadOnlyRootFiles	The file system is not set to read-only
	PodSetPullPolicyNotAlways	The mirror pull strategy is not always
	PodSetRunAsRootAllowed	Executed as a root account
	PodDangerousCapabilities	You have the dangerous option in capabilities such as ALL/SYS_ADMIN/NET_ADMIN
	PodlivenessProbeMissing	ReadinessProbe was not declared
	privilegeEscalationAllowed	Privilege escalation is allowed

Results Example

root@node1:/home/ubuntu/go/src/kubeye# ./ke audit --kubeconfig /home/ubuntu/config
NODENAME   SEVERITY   HEARTBEATTIME               REASON              MESSAGE
node18     danger     2020-11-19T10:32:03+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node19     danger     2020-11-19T10:31:37+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node2      danger     2020-11-19T10:31:14+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node3      danger     2020-11-27T17:36:53+08:00   KubeletNotReady     Container runtime not ready: RuntimeReady=false reason:DockerDaemonNotReady message:docker: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

NAME        SEVERITY   TIME                        MESSAGE
scheduler   danger     2020-11-27T17:09:59+08:00   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0      danger     2020-11-27T17:56:37+08:00   Get https://192.168.13.8:2379/health: dial tcp 192.168.13.8:2379: connect: connection refused

NAMESPACE        NODENAME                                EVENTTIME                   REASON                MESSAGE
default          node3.164b53d23ea79fc7                  2020-11-27T17:37:34+08:00   ContainerGCFailed     rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
default          node3.164b553ca5740aae                  2020-11-27T18:03:31+08:00   FreeDiskSpaceFailed   failed to garbage collect required amount of images. Wanted to free 5399374233 bytes, but freed 416077545 bytes
default          nginx-b8ffcf679-q4n9v.16491643e6b68cd7  2020-11-27T17:09:24+08:00   Failed                Error: ImagePullBackOff
default          node3.164b5861e041a60e                  2020-11-27T19:01:09+08:00   SystemOOM             System OOM encountered, victim process: stress, pid: 16713
default          node3.164b58660f8d4590                  2020-11-27T19:01:27+08:00   OOMKilling            Out of memory: Kill process 16711 (stress) score 205 or sacrifice child Killed process 16711 (stress), UID 0, total-vm:826516kB, anon-rss:819296kB, file-rss:0kB, shmem-rss:0kB
insights-agent   workloads-1606467120.164b519ca8c67416   2020-11-27T16:57:05+08:00   DeadlineExceeded      Job was active longer than specified deadline
kube-system      calico-node-zvl9t.164b3dc50580845d      2020-11-27T17:09:35+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system      kube-proxy-4bnn7.164b3dc4f4c4125d       2020-11-27T17:09:09+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system      nodelocaldns-2zbhh.164b3dc4f42d358b     2020-11-27T17:09:14+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29


NAMESPACE        NAME                      KIND         TIME                        MESSAGE
kube-system      node-problem-detector     DaemonSet    2020-11-27T17:09:59+08:00   [livenessProbeMissing runAsPrivileged]
kube-system      calico-node               DaemonSet    2020-11-27T17:09:59+08:00   [runAsPrivileged cpuLimitsMissing]
kube-system      nodelocaldns              DaemonSet    2020-11-27T17:09:59+08:00   [cpuLimitsMissing runAsPrivileged]
default          nginx                     Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing livenessProbeMissing tagNotSpecified]
insights-agent   workloads                 CronJob      2020-11-27T17:09:59+08:00   [livenessProbeMissing]
insights-agent   cronjob-executor          Job          2020-11-27T17:09:59+08:00   [livenessProbeMissing]
kube-system      calico-kube-controllers   Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing livenessProbeMissing]
kube-system      coredns                   Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing]   

Custom check

• Add custom npd rule methods

1. Deploy npd, ./ke add npd --kubeconfig /home/ubuntu/.kube/config
2. Ddit node-problem-detector-config configMap, such as: kubectl edit cm -n kube-system node-problem-detector-config
3. Add exception log information under the rule of configMap, rules follow regular expressions.

• Add custom best practice configuration

1. Use the -f parameter and file name config.yaml.
./ke audit -f /home/ubuntu/go/src/kubeye/examples/tmp/config.yaml --kubeconfig /home/ubuntu/.kube/config
2. config.yaml example, follow the JSON syntax.
ubuntu@node1:~/go/src/kubeye/examples/tmp$ cat config.yaml
checks:
  imageRegistry: warning

customChecks:
  imageRegistry:
    successMessage: Image comes from allowed registries
    failureMessage: Image should not be from disallowed registry
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          not:
            pattern: ^quay.io

ubuntu@node1:~/go/src/kubeye/examples/tmp$./ke audit -f /home/ubuntu/go/src/kubeye/examples/tmp/config.yaml
NAMESPACE     NAME                      KIND         TIME                        MESSAGE
default       nginx                     Deployment   2020-11-27T17:18:31+08:00   [imageRegistry]
kube-system   node-problem-detector     DaemonSet    2020-11-27T17:18:31+08:00   [livenessProbeMissing runAsPrivileged]
kube-system   calico-node               DaemonSet    2020-11-27T17:18:31+08:00   [cpuLimitsMissing runAsPrivileged]
kube-system   calico-kube-controllers   Deployment   2020-11-27T17:18:31+08:00   [cpuLimitsMissing livenessProbeMissing]
kube-system   nodelocaldns              DaemonSet    2020-11-27T17:18:31+08:00   [runAsPrivileged cpuLimitsMissing]
default       nginx                     Deployment   2020-11-27T17:18:31+08:00   [livenessProbeMissing cpuLimitsMissing]
kube-system   coredns                   Deployment   2020-11-27T17:18:31+08:00   [cpuLimitsMissing]