Kubernetes kind : Deployment terminationGracePeriodSeconds Alanı

Giriş

Kısaca tüm shutdown işleminin ne kadar sürmesi gerektiğini belirtir

Açıklaması şöyle

When the programmer or deployment pipeline executes the kubectl delete pod command, two procedures begin

1. Network rules coming into effect

- Kube-apiserver receives the pod deletion request and updates the state of the pod to Terminating at Etcd;

- Endpoint Controller deletes the IP of the pod from the Endpoint object;

- Kuber-proxy updates the rules of iptables according to the change of the Endpoint object, and no longer routes traffic to the deleted pod.

2. Deleting a pod

- Kube-apiserver receives the pod deletion request and updates the state of the pod to Terminating at Etcd

- Kubelet cleans up container-related resources at the node, such as storage, network

- Kubelet sends SIGTERM to the container; if there are no configurations for the process within the container, the container will exit at once.

- If the container didn’t exit within the default 30s, Kubelet will send SIGKILL and force it to exit.

Ancak bu iki işlem paralel çalışıyor, sıralı değil. Bu yüzden bazı hatalar oluşabiliyor. Açıklaması şöyle

- A pod that is currently in the middle of processing a request is removed, which, if the request is not idempotent, leads to an inconsistent state.

- Kubernetes routes traffic to pods that have already be en deleted, resulting in failure of processing requests and poor user experience.

Bu durumdan kaçınmak için graceful shutdown yapmak gerekir. Yani 

1. Pod network kurallarından silinir

2. Pod'un mevcut işlerini bitirmesi beklenir.

Pod network kurallarından silinmesinin önce gerçekleşmesi için Pod'un mevcut işlerini bitirmesini uzatmak gerekir. Bunun için preStop Hook kullanıır

Örnek

Şöyle yaparız. 

apiVersion: apps/v1
kind: Pod


metadata:
  name: my-pod
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - web
          containerPort : 80
      lifecycle
	  preStop:
          exec:
            command: ["sleep", "15"]

Pod açısından akış şöyle. Yani preHook süresinden sonra uygulamanın SIGTERM sinyalini yakalayıp sonlanması gerekiyor.

When the kubelet deletes a pod, it goes through the following steps:

- Triggers the preStop hook (if any).

- Sends the SIGTERM.

- Sends the SIGKILL signal (after 30 seconds).

Örnek

Şöyle yaparız. Burada

1. Önce Pod'un IP'sinin silinmesini beklemek için preStop Hook ile 10 saniye bekleniyor.

2. Daha sonra Spring uygulamasının shutdown endpoint çağrılıyor. SpringBoot uygulaması şöyle

server: 
  shutdown: graceful 

spring:  
  lifecycle:
    timeout-per-shutdown-phase: 30s

Yani 30 saniye içinde graceful shutdown yapmaya çalışıyor. Kubernetes açısından tüm shutdown işleminin 45 saniye sürmesini sağlamak için terminationGracePeriodSeconds 45 saniye yapılıyor.

kind: Deployment
apiVersion: apps/v1

metadata:
   name: gracefulshutdown-app
spec:
  replicas: 3
  selector:
     matchLabels:
           app: gracefulshutdown-app
  template:
    metadata:
       labels:
         app: gracefulshutdown-app
    spec:
      containers:
        - name: graceful-shutdown-test
          image: gracefulshutdown-app:latest
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]  #set prestop hook
       terminationGracePeriodSeconds: 45

Kubescape - Vulnerability Scanner

Bir yazı burada

Kubernetes API Server api EndPoint

Giriş

api ile biten endpoint sorgu kısa bir çıktı verir. Açıklaması şöyle

- While running in a Pod, the Kubernetes apiserver is accessible via a Service named kubernetes in the default namespace.

- Therefore, Pods can use the kubernetes.default.svc hostname to query the API server.

Örnek

Şöyle yaparız. Burada Pod içinden API Server'a erişilmeye çalışılıyor

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" ${APISERVER}/api

Açıklaması şöyle. Pod içindeki dizinlerde API Server'a erişmek için gerekli sertifikalar var. Bunları kullanmak gerekir

- The recommended way to authenticate to the API server is with a service account credential. By default, a Pod is associated with a service account, and a credential (token) for that service account is placed into the filesystem tree of each container in that Pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

- If available, a certificate bundle is placed into the filesystem tree of each container at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be used to verify the serving certificate of the API server.

- Finally, the default namespace to be used for namespaced API operations is placed in a file at /var/run/secrets/kubernetes.io/serviceaccount/namespace in each container.

Kubernetes kind: CronJob

Giriş

Açıklaması şöyle

spec > schedule: specify the cron expression for how frequently the container has to be invoked

containers > image: image name which was built in previous steps

successFulJobsHistoryLimit: keep the metadata related to the last ’n’ successful jobs for debugging or log monitoring

Örnek - veri tabanı yedekleme

Şöyle yaparız. 

schedule her gece çalışacağını belirtir

restartPolicy gerekirse tekrar başlayacağını belirtir

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-backup
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          volumes:
          - name: backup-volume
            hostPath:
              path: /mnt/backup
          containers:
          - name: db-backup
            image: mysql:5.7
            args:
            - mysqldump
            - --host=<database host>
            - --user=root
            - --password=<database password>
            - --result-file=/mnt/backup/all-databases.sql
            - <database name>
            volumeMounts:
            - name: backup-volume
              mountPath: /mnt/backup

Örnek

Şöyle yaparız

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reporting-job
spec:
  schedule: "0 8 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: reporting
              image: reportingservice:1.0.0
              imagePullPolicy: IfNotPresent
          restartPolicy: OnFailure

Şöyle yaparız

# get the cron job state
kubectl get cronjob

# for continuous monitoring
kubectl get jobs --watch

# to View logs 
# Replace "reporting-job-706356" with the job name in your system
pods=$(kubectl get pods \
  --selector=job-name=reporting-job-706356 \
  --output=jsonpath={.items[*].metadata.name})

kubectl logs $pods

Kubernetes Worker Üzerindeki Kube-Proxy - Node Üzerindeki Ağ Kurallarını Yönetir

1. Kube-Proxy Worker Node Üzerindeki Çalışır

Açıklaması şöyle. Kube-Proxy worker nod üzerinde çalışır.  

Kube proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

2. Kube-Proxy Worker Node Üzerindeki Ağ Kurallarını Yönetir

Açıklaması şöyle. Kube-Proxy network kurallarını iptables ile yönetir. 

Kube-Proxy listens for changes to Services and then updates the local IPTables or IPVS rules accordingly. This ensures that traffic is correctly routed to the appropriate pods in the cluster.

For example, suppose a Service is created in Kubernetes that maps to a set of pods with the label “app=myapp”. Kube-Proxy will create IPTables or IPVS rules that direct traffic to the appropriate pod based on the Service’s selector.

Şeklen şöyle

Açıklaması şöyle

Who is configuring those iptables rules?

It’s kube-proxy that collects endpoints from the control plane and maps service IP addresses to pod IPs (it also load balances the connections).

Kube-proxy is a DaemonSet that listens to changes to the Kubernetes API.

2. Load Balancing Yapar

Açıklaması şöyle

One of the tasks for the Kubernetes service is to load balance across these pods. To enable this, every node in a Kubernetes cluster runs a kube-proxy. kube-proxy is responsible for implementing a form of virtual IP for Services .

Kube-Proxy works in three modes — User Space, iptables, and IPVS. Kube-Proxy watches the Kubernetes control plane for the addition and removal of Service and Endpoint objects. It uses either of these modes to choose the backend pod. In userspace mode, it chooses a backend via a round-robin algorithm. In other modes, it's more of a random pick but they provide faster routing as they work in kernel space. You can read more on this here.

Why You Can't Ping a Kubernetes Service ?

Pod'a açılmış bir shell'den bir servisin IP adresini pinglersek cevap gelmediğini görürüz.

3. Kube-Proxy Kurulum

Açıklaması şöyle

Kube-Proxy usually runs in your cluster in the form of a DaemonSet. But it can also be installed directly as a Linux process on the node. This depends on your cluster installation type.

If you use kubeadm, it will install Kube-Proxy as a DaemonSet. If you manually install the cluster components using official Linux tarball binaries, it will run directly as a process on the node.

4. Kube-Proxy Modları

Bunlar şöyle

1 IPtables mode

Açıklaması şöyle

This is the default and most widely used mode today. In this mode Kube-Proxy relies on a Linux feature called IPtables. IPtables works as an internal packet processing and filtering component. It inspects incoming and outgoing traffic to the Linux machine. Then it applies specific rules against packets that match specific criteria.

2. IPVS mode

Açıklaması şöyle

IPVS is a Linux feature designed specifically for load balancing. This makes it a perfect choice for Kube-Proxy to use. In this mode, Kube-Proxy inserts rules into IPVS instead of IPtables.

...

Despite its advantages, IPVS might not be present in all Linux systems today. In contrast to IPtables which is a core feature of almost every Linux operating system.

3. KernelSpace mode

Açıklaması şöyle

This mode is specific to Windows nodes. In this mode Kube-proxy uses Windows Virtual Filtering Platform (VFP) to insert the packet filtering rules. The VFP on Windows works the same as IPtables on Linux, which means that these rules will also be responsible for rewriting the packet encapsulation and replacing the destination IP address with the IP of the backend Pod.

5. Kube-Proxy Modunu Görmek

Açıklaması şöyle

By default, Kube-proxy runs on port 10249 and exposes a set of endpoints that you can use to query Kube-proxy for information.

You can use the /proxyMode endpoint to check the kube-proxy mode.

Şöyle yaparız

curl -v localhost:10249/proxyMode.

6. Envoy - Kube-Proxy Alternatifi

Açıklaması şöyle

In addition to Kube-Proxy, another popular proxy used in Kubernetes is Envoy. Envoy is a high-performance proxy that provides advanced traffic management and load-balancing capabilities. Envoy can be used as a replacement for Kube-Proxy to implement Kubernetes Services or can be used as an independent component to provide advanced traffic management features.

Envoy is used in many production environments and can provide benefits such as advanced load-balancing algorithms, circuit breaking, and distributed tracing.

However, Envoy requires additional setup and configuration compared to Kube-Proxy, and may not be compatible with all network environments. Additionally, Envoy is generally used in more complex scenarios, such as multi-cluster or multi-cloud environments, and may be overkill for simpler use cases.

Secret'i Volume Olarak Kullanma

Giriş

1. volume ile Secret volume haline getirilir

2. Pod volumeMounts ile bu volume'u yükler.

Örnek

Şöyle yaparız

apiVersion: v1
stringData:
  file.conf: |-
     username=demo
     password=my_plain_password
kind: Secret
metadata:
  name: my_secret
type: Opaque
---
apiVersion: v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  containers:
     ...
     volumeMounts:
     - name: secret-file
       mountPath: "path/in/the/pod/where/to/mount/the/file"
       subPath: file.conf # Just the file to mount
volumes:
  - name: secret-file
  secret:
     secretName: my_secret # same as secret's metadata name

Örnek

Şöyle yaparız. Burada secret veri bir volume'a yükleniyor. Her Key/Value çifti ayrı bir dosya

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=
  password: MTIzNDU2
--
apiVersion: v1
kind: Pod
metadata:
  name: basic-app
spec:
  volumes:
    - name: my-volume-for-secret
      secret:
        secretName: my-secret
  containers:
    - name: basic-app
      image: nginx
      volumeMounts:
        - name: my-volume-for-secret
          mountPath: /etc/my-secret-vol
          readOnly: true

Secret veriye erişmek için şöyle yaparız

> kubectl exec basic-app -- ls /etc/my-secret-vol
password 
username

> kubectl exec basic-app — cat /etc/my-secret-vol/username
admin

> kubectl exec basic-app — cat /etc/my-secret-vol/password
123456

ConfigMap'i Volume Olarak Kullanma

Giriş

1. volume ile ConfigMap volume haline getirilir

2. Pod volumeMounts/mountPath ile bu volume'u yükler. 

3. volumeMounts/mountPath bir dizin ismi ise ConfigMap'teki her data satırı ayrı bir dosya gibidir

Örnek - Çoklu Dosya

Şöyle yaparız. Burada ConfigMap bir volume'a yükleniyor. Her Key/Value satırı ayrı bir dosya gibidir.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  env: prod
  welcomeMessage: "Hello, welcome to kubernetes in a nutshell"
--
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  volumes:
    - name: my-volume
      configMap:
        name: my-configmap
  containers:
    - name: basic-app
      image: nginx
      volumeMounts:
        - name: my-volume
          mountPath: /etc/name

Görmek için şöyle yaparız

> kubectl exec my-app — cat /etc/name/env
prod

> kubectl exec my-app — cat /etc/name/welcomeMessage
Hello, welcome to kubernetes in a nutshell

Örnek - Çoklu Dosya

Elimizde şöyle bir dizim olsun

root/config-files/
                 |- user-data.txt
                 |- admin-info.txt


# user-data.txt
username: superuser
password: admin123

# admin-info.txt
city: Nobeoka
state: Miyazaki
country: Japan

ve tüm dosyaları ConfigMap yapalım

# Configmap with directory
k create configmap user-config --from-file=/root/config-files

ConfigMap'e bakalım. Çıktısı şöyle. Burada user-config bir volume ve içinde de iki tane dosya var

> k get configmap user-config -o yaml

apiVersion: v1
data:
  admin-info.txt: |
    username: superuser
    password: admin123
  user-data.txt: |
    city: Nobeoka
    state: Miyazaki
    country: Japan
kind: ConfigMap
metadata:
  creationTimestamp: "2022-08-07T09:38:22Z"
  name: user-config
  namespace: default
  resourceVersion: "2007"
  uid: 915e805a-cb55-4309-977a-566b7a8ed6ac

Volume olarak yüklemek için şöyle yaparız

# Pod-definition with configmap mounted as a volume into the pod
apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
  - name: wordpress
    image: wordpress
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config  # Directory where files will be mounted
  volumes:
    - name: config-volume
      configMap:
        name: user-config

Artık pod'un /etc/config dizininde admin-info.txt ve user-data.txt isimli iki tane dosya var. Eğer tüm dosyaları değil de sadece bazı dosyaları kullanmak istersek şöyle yaparız. Burada items ile ConfigMap üzerindeki dosya ismi ve Pod üzerindeki ismi belirtiliyor

# Pod-definition with configmap. Importing only the necessary files into the pods. 
# e.g: user-data.txt

apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
    - name: wordpress
      image: wordpress
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: user-config
        items:
        - key: user-data.txt  # filename on configmap
          path: pod-user-data.txt # filename on pod

Örnek - Tek Dosya

Şöyle yaparız. Burada file.conf isimli tek bir dosya var. subPath ile volume üzerindeki dosya belirtilir. mountPath ile de dosyanın pod üzerindeki yeri belirtilir. Normalde subPath'e gerek yok

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config-map
data:
  file.conf: |
     param1=value1
     param2=value2
     paramN=valueN
---
apiVersion: v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  containers:
     ...
     volumeMounts:
     - name: config
       mountPath: "path/in/the/pod/where/to/mount/the/file"
       subPath: file.conf
volumes:
  - name: config
  configMap:
    name: my-config-map
    items:
    - key: "file.conf" # filename on configmap
      path: "file.conf" # filename on pod