Kubernetes kind : Deployment terminationGracePeriodSeconds Alanı

Giriş

Kısaca tüm shutdown işleminin ne kadar sürmesi gerektiğini belirtir

Açıklaması şöyle

When the programmer or deployment pipeline executes the kubectl delete pod command, two procedures begin

1. Network rules coming into effect

- Kube-apiserver receives the pod deletion request and updates the state of the pod to Terminating at Etcd;

- Endpoint Controller deletes the IP of the pod from the Endpoint object;

- Kuber-proxy updates the rules of iptables according to the change of the Endpoint object, and no longer routes traffic to the deleted pod.

2. Deleting a pod

- Kube-apiserver receives the pod deletion request and updates the state of the pod to Terminating at Etcd

- Kubelet cleans up container-related resources at the node, such as storage, network

- Kubelet sends SIGTERM to the container; if there are no configurations for the process within the container, the container will exit at once.

- If the container didn’t exit within the default 30s, Kubelet will send SIGKILL and force it to exit.

Ancak bu iki işlem paralel çalışıyor, sıralı değil. Bu yüzden bazı hatalar oluşabiliyor. Açıklaması şöyle

- A pod that is currently in the middle of processing a request is removed, which, if the request is not idempotent, leads to an inconsistent state.

- Kubernetes routes traffic to pods that have already be en deleted, resulting in failure of processing requests and poor user experience.

Bu durumdan kaçınmak için graceful shutdown yapmak gerekir. Yani 

1. Pod network kurallarından silinir

2. Pod'un mevcut işlerini bitirmesi beklenir.

Pod network kurallarından silinmesinin önce gerçekleşmesi için Pod'un mevcut işlerini bitirmesini uzatmak gerekir. Bunun için preStop Hook kullanıır

Örnek

Şöyle yaparız. 

apiVersion: apps/v1
kind: Pod


metadata:
  name: my-pod
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - web
          containerPort : 80
      lifecycle
	  preStop:
          exec:
            command: ["sleep", "15"]

Pod açısından akış şöyle. Yani preHook süresinden sonra uygulamanın SIGTERM sinyalini yakalayıp sonlanması gerekiyor.

When the kubelet deletes a pod, it goes through the following steps:

- Triggers the preStop hook (if any).

- Sends the SIGTERM.

- Sends the SIGKILL signal (after 30 seconds).

Örnek

Şöyle yaparız. Burada

1. Önce Pod'un IP'sinin silinmesini beklemek için preStop Hook ile 10 saniye bekleniyor.

2. Daha sonra Spring uygulamasının shutdown endpoint çağrılıyor. SpringBoot uygulaması şöyle

server: 
  shutdown: graceful 

spring:  
  lifecycle:
    timeout-per-shutdown-phase: 30s

Yani 30 saniye içinde graceful shutdown yapmaya çalışıyor. Kubernetes açısından tüm shutdown işleminin 45 saniye sürmesini sağlamak için terminationGracePeriodSeconds 45 saniye yapılıyor.

kind: Deployment
apiVersion: apps/v1

metadata:
   name: gracefulshutdown-app
spec:
  replicas: 3
  selector:
     matchLabels:
           app: gracefulshutdown-app
  template:
    metadata:
       labels:
         app: gracefulshutdown-app
    spec:
      containers:
        - name: graceful-shutdown-test
          image: gracefulshutdown-app:latest
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]  #set prestop hook
       terminationGracePeriodSeconds: 45

Kubescape - Vulnerability Scanner

Bir yazı burada

Kubernetes API Server api EndPoint

Giriş

api ile biten endpoint sorgu kısa bir çıktı verir. Açıklaması şöyle

- While running in a Pod, the Kubernetes apiserver is accessible via a Service named kubernetes in the default namespace.

- Therefore, Pods can use the kubernetes.default.svc hostname to query the API server.

Örnek

Şöyle yaparız. Burada Pod içinden API Server'a erişilmeye çalışılıyor

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" ${APISERVER}/api

Açıklaması şöyle. Pod içindeki dizinlerde API Server'a erişmek için gerekli sertifikalar var. Bunları kullanmak gerekir

- The recommended way to authenticate to the API server is with a service account credential. By default, a Pod is associated with a service account, and a credential (token) for that service account is placed into the filesystem tree of each container in that Pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

- If available, a certificate bundle is placed into the filesystem tree of each container at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be used to verify the serving certificate of the API server.

- Finally, the default namespace to be used for namespaced API operations is placed in a file at /var/run/secrets/kubernetes.io/serviceaccount/namespace in each container.

Kubernetes kind: CronJob

Giriş

Açıklaması şöyle

spec > schedule: specify the cron expression for how frequently the container has to be invoked

containers > image: image name which was built in previous steps

successFulJobsHistoryLimit: keep the metadata related to the last ’n’ successful jobs for debugging or log monitoring

Örnek - veri tabanı yedekleme

Şöyle yaparız. 

schedule her gece çalışacağını belirtir

restartPolicy gerekirse tekrar başlayacağını belirtir

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-backup
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          volumes:
          - name: backup-volume
            hostPath:
              path: /mnt/backup
          containers:
          - name: db-backup
            image: mysql:5.7
            args:
            - mysqldump
            - --host=<database host>
            - --user=root
            - --password=<database password>
            - --result-file=/mnt/backup/all-databases.sql
            - <database name>
            volumeMounts:
            - name: backup-volume
              mountPath: /mnt/backup

Örnek

Şöyle yaparız

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reporting-job
spec:
  schedule: "0 8 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: reporting
              image: reportingservice:1.0.0
              imagePullPolicy: IfNotPresent
          restartPolicy: OnFailure

Şöyle yaparız

# get the cron job state
kubectl get cronjob

# for continuous monitoring
kubectl get jobs --watch

# to View logs 
# Replace "reporting-job-706356" with the job name in your system
pods=$(kubectl get pods \
  --selector=job-name=reporting-job-706356 \
  --output=jsonpath={.items[*].metadata.name})

kubectl logs $pods