Kubernetes PV ve PVC AccessModes Uyumu

Giriş

Şeklen şöyle

Açıklaması şöyle

ReadWriteOnce (RWO): This is the default access mode. It allows the PVC to be mounted as read-write by a single node in the cluster. This means that the PVC can be used by a single pod running on that node and is not available to other nodes in the cluster.

ReadOnlyMany (ROX): This access mode allows the PVC to be mounted as read-only by many nodes in the cluster. This means that the PVC can be used by multiple pods running on different nodes, but they can only read from it, not write to it.

ReadWriteMany (RWX): This access mode allows the PVC to be mounted as read-write by many nodes in the cluster. This means that the PVC can be used by multiple pods running on different nodes, and they can both read from and write to it.

ReadWriteOncePod: The ReadWriteOncePod storage class is a pre-defined storage class that can be used to create a persistent volume with ReadWriteOnce access mode that is intended to be used by a single pod. Kubernetes ensures that pod is the only pod across your whole cluster that can read that PVC or write to it.

Örnek

3 tane PV şöyle olsun. Hepsi hostPath kullanıyor. 

1. ReadOnlyMany

2. ReadWriteMany

3. ReadWriteOnce

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-rox
spec:
  storageClassName: standard
  capacity:
    storage: 5Gi
  accessModes:
    - ReadOnlyMany
  hostPath:
    path: /data/my-pv-rox

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-rwx
spec:
  storageClassName: standard
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /data/my-pv-rwx

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-rwo
spec:
  storageClassName: standard
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/my-pv-rwo

Bunlarla uyumlu 3 tane PVC yaratmak için şöyle yaparız.  Hepsinin storageClassName ismi standard ve accessModes şöyle

1. ReadOnlyMany

2. ReadWriteMany

3. ReadWriteOnce

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-rox
spec:
  storageClassName: standard
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-rwx
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-rwo
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Common Pods Errors - Startup errors

Giriş

Bazı hatalar şöyle

1. ImagePullBackoff
2. ImageInspectError
3. ErrImagePull
4. ErrImageNeverPull
5. RegistryUnavailable
6. InvalidImageName

Ayrıca FailedScheduling de çok yaygın

Ne Yapmak Lazım

1. kubectl describe pod $POD_NAME | less ile duruma bakılabilir

2. Eğer bir şeyi değiştirip tekrar denemek istersek şöyle yaparız

kubectl edit pod $POD_NAME
# or 
kubectl edit deployment $DEPLOYMENT_NAME

# or if you just want to view the information
kubectl get deployment $DEPLOYMENT_NAME -oyaml

3. loglara bakabiliriz. Şöyle yaparız

kubectl logs $POD_NAME
kubectl logs $POD_NAME -c $CONTAINER_NAME

# or if you want to follow continuous output
kubectl logs -f $POD_NAME

# Another super useful debugging tool is the -p/--previous flag, 
# which you can use in the case that an instance keeps crashing/ 
# there was an unexpected restart.
kubectl logs -p $POD_NAME

ContainerCreating 

Açıklaması şöyle

Generally the pod will be stuck in states like ContainerCreating or if there’s an image issue you might see ImagePullBackoff , possibly indicating the image couldn’t be found or authenticated to.

ImagePullBackOff 

Açıklaması şöyle

This error means K8s is unable to pull the image for one of the containers in the Pod.

Common cause of the error could be one of the following,

1. The Image name is invalid
2. You specified non existing tag for the Image
4. The image that you are trying to pull belongs to a private registry and k8s does not have credentials to access it.

The first two cases can be solved by correcting the image name and tag.

For the last, you should add the credentials to your private registry in a Secret and reference it in your Pods.

FailedScheduling 

nodeSelector veya nodeAffinity kullanılıyorsak olabilir. 

Örnek

kubectl -n <namespace> describe pod <podname> çıktısı şöyledir

Warning FailedScheduling 118s (x26 over 7m43s) default-scheduler 0/53 nodes are available: 53 node(s) didn’t match Pod’s node affinity

kind: DaemonSet

Giriş

Açıklaması şöyle. Yani DaemonSet her worker node üzerinde çalışır

DaemonSets are great for running a single instance of an application on every node in the cluster. These are applications that need to be run on every node in the cluster. This could be things like logging or monitoring agents. For example, if you have a logging agent that you want to run on every node in your cluster, you could use a DaemonSet to make sure that there is an instance of the agent running on each node. This is useful because it ensures that the same instance of the application is running on each node, which can be important for tasks that require node-level access or coordination.

Örnek - Logging Agent

Şöyle yaparız

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logging-agent-daemonset
spec:
  selector:
    matchLabels:
      app: logging-agent
  template:
    metadata:
      labels:
        app: logging-agent
    spec:
      containers:
      - name: logging-agent
        image: my-logging-agent-image:latest
        volumeMounts:
        - name: logs
          mountPath: /var/log
      volumes:
      - name: logs
        hostPath:
          path: /var/log

Kubernetes kind : Deployment terminationGracePeriodSeconds Alanı

Giriş

Kısaca tüm shutdown işleminin ne kadar sürmesi gerektiğini belirtir

Açıklaması şöyle

When the programmer or deployment pipeline executes the kubectl delete pod command, two procedures begin

1. Network rules coming into effect

- Kube-apiserver receives the pod deletion request and updates the state of the pod to Terminating at Etcd;

- Endpoint Controller deletes the IP of the pod from the Endpoint object;

- Kuber-proxy updates the rules of iptables according to the change of the Endpoint object, and no longer routes traffic to the deleted pod.

2. Deleting a pod

- Kube-apiserver receives the pod deletion request and updates the state of the pod to Terminating at Etcd

- Kubelet cleans up container-related resources at the node, such as storage, network

- Kubelet sends SIGTERM to the container; if there are no configurations for the process within the container, the container will exit at once.

- If the container didn’t exit within the default 30s, Kubelet will send SIGKILL and force it to exit.

Ancak bu iki işlem paralel çalışıyor, sıralı değil. Bu yüzden bazı hatalar oluşabiliyor. Açıklaması şöyle

- A pod that is currently in the middle of processing a request is removed, which, if the request is not idempotent, leads to an inconsistent state.

- Kubernetes routes traffic to pods that have already be en deleted, resulting in failure of processing requests and poor user experience.

Bu durumdan kaçınmak için graceful shutdown yapmak gerekir. Yani 

1. Pod network kurallarından silinir

2. Pod'un mevcut işlerini bitirmesi beklenir.

Pod network kurallarından silinmesinin önce gerçekleşmesi için Pod'un mevcut işlerini bitirmesini uzatmak gerekir. Bunun için preStop Hook kullanıır

Örnek

Şöyle yaparız. 

apiVersion: apps/v1
kind: Pod


metadata:
  name: my-pod
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - web
          containerPort : 80
      lifecycle
	  preStop:
          exec:
            command: ["sleep", "15"]

Pod açısından akış şöyle. Yani preHook süresinden sonra uygulamanın SIGTERM sinyalini yakalayıp sonlanması gerekiyor.

When the kubelet deletes a pod, it goes through the following steps:

- Triggers the preStop hook (if any).

- Sends the SIGTERM.

- Sends the SIGKILL signal (after 30 seconds).

Örnek

Şöyle yaparız. Burada

1. Önce Pod'un IP'sinin silinmesini beklemek için preStop Hook ile 10 saniye bekleniyor.

2. Daha sonra Spring uygulamasının shutdown endpoint çağrılıyor. SpringBoot uygulaması şöyle

server: 
  shutdown: graceful 

spring:  
  lifecycle:
    timeout-per-shutdown-phase: 30s

Yani 30 saniye içinde graceful shutdown yapmaya çalışıyor. Kubernetes açısından tüm shutdown işleminin 45 saniye sürmesini sağlamak için terminationGracePeriodSeconds 45 saniye yapılıyor.

kind: Deployment
apiVersion: apps/v1

metadata:
   name: gracefulshutdown-app
spec:
  replicas: 3
  selector:
     matchLabels:
           app: gracefulshutdown-app
  template:
    metadata:
       labels:
         app: gracefulshutdown-app
    spec:
      containers:
        - name: graceful-shutdown-test
          image: gracefulshutdown-app:latest
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]  #set prestop hook
       terminationGracePeriodSeconds: 45

Kubescape - Vulnerability Scanner

Bir yazı burada

Kubernetes API Server api EndPoint

Giriş

api ile biten endpoint sorgu kısa bir çıktı verir. Açıklaması şöyle

- While running in a Pod, the Kubernetes apiserver is accessible via a Service named kubernetes in the default namespace.

- Therefore, Pods can use the kubernetes.default.svc hostname to query the API server.

Örnek

Şöyle yaparız. Burada Pod içinden API Server'a erişilmeye çalışılıyor

# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc

# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

# Read this Pod's namespace
NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)

# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt

# Explore the API with TOKEN
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" ${APISERVER}/api

Açıklaması şöyle. Pod içindeki dizinlerde API Server'a erişmek için gerekli sertifikalar var. Bunları kullanmak gerekir

- The recommended way to authenticate to the API server is with a service account credential. By default, a Pod is associated with a service account, and a credential (token) for that service account is placed into the filesystem tree of each container in that Pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

- If available, a certificate bundle is placed into the filesystem tree of each container at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be used to verify the serving certificate of the API server.

- Finally, the default namespace to be used for namespaced API operations is placed in a file at /var/run/secrets/kubernetes.io/serviceaccount/namespace in each container.

Kubernetes kind: CronJob

Giriş

Açıklaması şöyle

spec > schedule: specify the cron expression for how frequently the container has to be invoked

containers > image: image name which was built in previous steps

successFulJobsHistoryLimit: keep the metadata related to the last ’n’ successful jobs for debugging or log monitoring

Örnek - veri tabanı yedekleme

Şöyle yaparız. 

schedule her gece çalışacağını belirtir

restartPolicy gerekirse tekrar başlayacağını belirtir

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-backup
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          volumes:
          - name: backup-volume
            hostPath:
              path: /mnt/backup
          containers:
          - name: db-backup
            image: mysql:5.7
            args:
            - mysqldump
            - --host=<database host>
            - --user=root
            - --password=<database password>
            - --result-file=/mnt/backup/all-databases.sql
            - <database name>
            volumeMounts:
            - name: backup-volume
              mountPath: /mnt/backup

Örnek

Şöyle yaparız

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reporting-job
spec:
  schedule: "0 8 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: reporting
              image: reportingservice:1.0.0
              imagePullPolicy: IfNotPresent
          restartPolicy: OnFailure

Şöyle yaparız

# get the cron job state
kubectl get cronjob

# for continuous monitoring
kubectl get jobs --watch

# to View logs 
# Replace "reporting-job-706356" with the job name in your system
pods=$(kubectl get pods \
  --selector=job-name=reporting-job-706356 \
  --output=jsonpath={.items[*].metadata.name})

kubectl logs $pods