Common Pods Errors - CrashLoopBackOff

Giriş

Açıklaması şöyle

If the container can’t start, then K8s shows the CrashLoopBackOff message as a status.

Usually, a container can’t start when:

1. There’s an error in the application that prevents it from starting.

2. You misconfigured the container.

3. The Liveness probe failed too many times.

4. You should try and retrieve the logs from that container to investigate why it failed.

If you can’t see the logs because your container is restarting too quickly, you can use the following command:

Önce CrashLoopBackOff olan Pod olup olmadığını görmek için şöyle yaparız. Evet bir tane var

kubectl get pods -n <namespace>

NAME                     READY     STATUS             RESTARTS   AGE
nginx-5796d5bc7d-xtl6q   0/1       CrashLoopBackOff   4          1m

CrashLoopBackOff  Olmasaydı

Açıklaması şöyle. Yani kısaca gereksiz yere çok fazla kayna tüketilecekti

- In the absence of CrashLoopBackOff, Kubernetes would try to restart the container right after it crashes.

- This could lead to a significant number of restart attempts within a short span of time, thereby putting unnecessary strain on the system.

- The increased failure rate could affect the availability of the application running inside the container.

Neden CrashLoopBackOff Olduğunu Görmek İçin Kullanılabilecek Komutlar Şöyle

1. kubectl describe pod

2. kubectl logs

3. kubectl get events

1. kubectl describe pod

Burada Last State altındaki Reason alanına bakmak gerekir. 

Reason Error İse

Örnek  - Yetersiz Heap

"Caused by: java.lang.OutOfMemoryError: Java heap space" şöyledir

Containers:
  heapkiller:
    ....
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
...
Events:
  Type     Reason     Age                From       Message
  ----     ------     ----               ----       -------
...
  Warning  BackOff    7s (x7 over 89s)   kubelet    Back-off restarting failed container

Reason OOMKilled İse

Örnek - Kubernetes Killed The Pod

Şöyledir. OOMKilled Kubernetes, Pod'u sınırları aştığı için öldürdü anlamına gelir.

Containers:
  heapkiller:
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
Events:
  Type     Reason     Age                  From       Message
 ----     ------     ----                 ----        ------
...  
...
 Warning  BackOff    6s (x7 over 107s)    kubelet     Back-off restarting failed container

Status Evicted

Örnek

Şöyledir. SystemOOM hatası da görülür

~ kubectl describe pod/heapkiller

Status:           Failed
Reason:           Evicted
Message:          The node was low on resource: memory.
Containers:
  heapkiller:
    State:          Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was terminated
      Exit Code:    137
      Reason:       OOMKilled

2. kubectl logs

Şöyle yaparız -p ile bir önceki hatalı Pod'un logları görülebilir. Veya bazen Pod içindeki bir container'a bakmak gerekir.

kubectl logskubectl logs <pod name> -n <namespace> -p
kubectl logs <pod name> -n <namespace> --previous
kubectl logs <pod name> -n <namespace> --all-containers
kubectl logs <pod name> -n <namespace> -c mycontainer
kubectl logs <pod name> -n <namespace> --tail 50