Observability in Kubernetes

Credits: https://www.elastic.co/

We use dockers to run applications in a separate environment with all its dependencies.
As the number of apps grew, number of dockers grew too.
So there came a concept of container orchestration.
Which led to evolution of tools like docker swarm, Kubernetes. And by far Kubernetes is the most widely used tool.

So what is the main USP of container orchestration?
It helps us to monitor huge number of docker containers and thereafter take certain actions like start, stop, scale-up, scale-down, etc.
Before even taking action, it is essential for a container orchestration tool to ‘observe’ the changes.

So how does Kubernetes does that? It uses these concepts:
1. Probes: Readiness, Liveness and Startup
2. Container logging
3. Monitoring: Third party integrations

Probes: Readiness, Liveness

Readiness probes helps Kubernetes identify if the pod is up correctly, all its dependencies are up and pod is in a state to serve requests. K8s reflects the same in pod status (Ready)

Liveness probes helps Kubernetes identify if the pod is up . But it does not ensure if pod is ready to serve requests.

If readiness probe fails, the service object does not redirect requests to those failed pods, but the pod remains as it is.

If liveness probe fails, the pod itself gets restarted.

There are various ways to configure probes: HttpGet, TCP and Exec.
Here is an example how to configure readiness & liveness probe using HttpGet

apiVersion: v1
kind: Pod
metadata:
 name: simple-webapp
 labels:
 name: simple-webapp
spec:
 containers:
 — name: simple-webapp
 image: simple-webapp
 ports:
 — containerPort: 8080
 livenessProbe:
  httpGet: 
      path: /api/healthy
      port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 8
 readinessProbe:
  httpGet: 
      path: /api/ready
      port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 8

Container logging

1. In order to view logs of a container, use below:

kubectl logs -f <pod-name>` (f option is to stream logs live).

2. If multiple containers are running in a single pod, it requires the container name as well, else the command would fail:

`kubectl logs -f <pod-name> <container-name>`

Monitor

What to Monitor:

1. Count of nodes & Pods.
2. CPU usage.
3. Memory consumed.
4. n/w bandwidth
5. Disk utilisation of nodes and pods.

Tools to integrate with k8s for monitoring metrics:

1. Metrics server
2. Prometheus
3. Elastic stack
4. Datadog
5. Dynatrace

Simplest way to setup a Metrics server in your cluster

1. git clone https://lnkd.in/dmp38pfZ — download the deployment binaries
   2. cd kubernetes-metrics-server
   3. kubectl create -f . -> creates set of pods, services and roles to enable metric server to poll for performance metrics of cluster
   4. kubectl top node — to view the metrics of nodes
   5. kubectl top pod — to view the metrics of pods

Folks, if you like my content, would you consider following me on linked in at: https://www.linkedin.com/in/hitesh-pattanayak-52290b160/