Kubernetes

Kubernetes (K8s): a orchastration tool for containers in a cluster

Because the word "Kubernetes" is hard to spell right, and there are 8 letters "ubernete" between "K" and "s", "K8s" abbreviation is used.

There are many other tools in container ecosystems:

Cloudnative Trailmap

Cloudnative Trailmap

Cloudnative Landscape

Cloudnative Landscape

Installation

Note that the hostname of a node cannot contain upper case letters to use Kubernetes. Use hostnamectl set-hostname to change hostname.

Add the following lines to /etc/docker/daemon.json:

{
    "insecure-registries" : ["localhost:32000"]
}

and then restart docker with: sudo systemctl restart docker

Install Microk8s

Assuming you have Ubuntu 22.04:

#!/bin/sh

# make sure docker is installed and we have permission
sudo usermod -a -G docker $USER
newgrp docker

# install microk8s to run locally
# stable version of 1.27
sudo snap install microk8s --classic --channel=1.27/stable
# setup user group
sudo usermod -a -G microk8s $USER
# refresh permission group
newgrp microk8s
# give access
sudo chown -f -R $USER ~/.kube
# check installation
microk8s status --wait-ready

# now install kubectl, although microk8s has its own, we will use system's kubectl
sudo apt-get install -y ca-certificates curl apt-transport-https
# get key
sudo mkdir /etc/apt/keyrings/
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
# add repo
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
# install kubectl
sudo apt-get update && sudo apt-get install -y kubectl

# now we need to connect kubectl with microk8s
# this might need to be reset when your network status changes
# by making ~/.kube/config
microk8s config > ~/.kube/config
# should display correctly
kubectl cluster-info

# install helm
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install -y apt-transport-https
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install -y helm

Install K9s

Go to Here and download the k9s_Linux_amd64.tar.gz file, and put the file into /usr/local/bin. That's it.

Basic Concepts

Master Node: manage Cluster Network, can have multiple master nodes

Worker Nodes: actual work, can have multiple containers in one node

Starting V1.19, Moby isn't needed as you will use crictl to run command directly in a node. Docker, before version 20.10, has a container runtime, but they started to use containerd as their underlying container runtime for better modularity.

Node Pool: a group of virtual machines of the same size

Application: e.g. docker, database

Cluster IP: virtual IP address in Cluster Network

Pod: an abstraction of containers (so you don't have to always use docker), ephemeral

ConfigMap & Secrets: like a per-cluster .env file

Volume: a storage (either local or remote) that is attached to a Pod for consistent data storage like database (because container themselves can't store persistent data)

Kubernetes doesn't manage data persistance

Deployments: blueprint for how to create a kind of Pods at scale.

Don't use Deployments to create Pods that is stateful (e.g. database) and need global consistency. Use StatefulSet instead.

StatefulSet: DB replication, scaling, synchronized read and writes.

In practice, don't use StatefulSet. You probably want to use 3rd party service for that.

Configuration

Master can be accessed through

Configuration has 3 components

We directly use yaml or json configuration to launch Service, Deployment (which creates Pods), Secrets, and ConfigMap.

Use kubectl get all, kubctl get secret, and kubctl get configmap to see all launched Components. Use kubctl get node -o wide to see info of nodes.

You can generate .yaml config by using kubectl --dry-run=client -o yaml

Configurations are hard to write correct, here are some great references:

Learn more about configurations Here

Cluster Simulation

Tools:

KubeCtl

Context: which cluster you are working with and metadata associated to that context. Information stored in ~/.kube/config

Context Commands

kubectl config current-context # show which context is used currently
kubectl config get-context # show all context avaliable
kubectl config use-context [contextName] # switch to context
kubectx # a short name for "kubectl config"
kubectl config rename-context [oldName] [newName]
kubectl config delete-context [contextName]

Namespace Commands

kubectl get namespace # List all Namespace
kubectl get ns # List all Namespace
kubectl config set-context --current --namespace=[namespaceName] # switch to Namespace, all commands will send to that Namespace
kubectl create namespace [namespaceName]
kubectl delete namespace [namespaceName] # delete Namespace and all related Components
kubectl get pods --all-namespaces # list all pods in all Namespaces
kubectl get pods --namespace=[namespaceName] # list all pods in specific Namespace
kubectl get pods -n [namespaceName] # list all pods in specific Namespace

You can put namespace on any Component under the metadata section (in addition to name), so you can delete them in a batch. (Namespace is on its own a Component)

Labels: self-defined tags for each Components, work with Selectors Selectors: use labels to filter or select Components, used to refer to Components (also you can pass --selector=[name] to get commands to filter stuff)

Pods Commands

kubectl create -f [fileName.yaml] # create a pod
kubectl get pods -o wide # print all pods
kubectl describe pod [podName] # show pod info
kubectl get pod [podName] -o yaml # extract pod definition in yaml
kubectl exec -it [podName] -- sh # launch interactive `sh` shell in a pod
kubectl exec -it [podName] -c [containerName] -- sh # for multi-container pod
kubectl logs [podName] -c [containerName] # getting logs for a container
kubectl delete -f [fileName.yaml] # delete a pod
kubectl delete pod [podName] # delete a pod in 30s
kubectl delete pod [podName] --wait=false # delete a pod with no wait
kubectl delete pod [podName] --grace-period=0 --force # force delete a pod
kubectl port-forward [kind]/[serviceName] [podPort]:[localPort] # ssh forward service

Node Commands

kubectl get nodes # see all Nodes
kubectl describe nodes [nodeName] # see status of a Node

Pods

Pod State:

Init Containers: executable logics to run before actual app starts to run (install dependencies)

Networking

IP Address

Workload

Workload: a Component that does heavy work, here are the inheritage

Since setting up database is hard and inefficient, I would prefer not to setup a 3rd party cloud service. So nothing is covered on this tutorial.

ReplicaSets Commands

kubectl get rs # list ReplicaSets
kubectl describe rs [replicaSetName]
kubectl delete -f [fileName.yaml]
kubectl delete rs [replicaSetName]

Deployment:

Deployment Commands

kubectl get rs # list ReplicaSets
kubectl get deploy # list ReHTTPplicaSets
kubectl describe deploy [deploymentName]
kubectl delete -f [fileName.yaml]
kubectl delete deploy [deploymentName]

kubectl apply -f [fileName.yaml] # to update a deployment (not `create`)
kubectl rolllout status # get the progress of the update
kubectl rollout history deployment [deploymentName] # get history
kubectl rollout undo [deploymentName] # rollback a deployment
kubectl rollout undo [deploymentName] --to-revision=[revisionNumber] # rollback to a revision

DaemonSet Commands

kubectl get ds # list ReplicaSets
kubectl describe ds [daemonSetName]
kubectl delete -f [fileName.yaml]
kubectl delete ds [daemonSetName]

Job Commands

kubectl get job # list ReplicaSets
kubectl describe job [jobName]
kubectl delete -f [fileName.yaml]
kubectl delete job [jobName]

CronJob Commands

kubectl get cj # list ReplicaSets
kubectl describe cj [cronJobName]
kubectl delete -f [fileName.yaml]
kubectl delete cj [cronJobName]

Blue-Green Deployments

Blue: Pods in production v1 Green: Pods in new version v2

Blue-Green Deployment: when everything is ready, switch selector of Service from v1 to v2 (switch blue and green)

Service

Service: for both external and internal communication

Service Commands

kubectl apply -f [fileName.yaml] # to deploy a service
kubectl get svc # list ReplicaSets
kubectl describe svc [serviceName]
kubectl delete -f [fileName.yaml]
kubectl delete svc [serviceName]

Volumes

Volumes: cluster-wide storage system outside of cluster (provided by 3rd party service), vendors create plugins according to Container Storage Interface

Types of Persistent Volume

Types of Persistent Volume

Reclaim Policies:

Access Modes:

States:

ConfigMap

ConfigMap: Externalize Environment Configuration as a Component, can create from

To see the changes, containers have to restart and K8s will inject the environmental value. You could mount Volume in ConfigMap to make it non-static, but you need to read from files.

ConfigMap Commands

kubectl apply -f [fileName.yaml] # to deploy a ConfigMap
kubectl create cm [configMapName] --from-file=[fileName.txt] # create imperatively from file
kubectl create cm [configMapName] --from-file=[directory/] # create imperatively from directory
kubectl get cm
kubectl get cm [configMapName] -o YAML # save configMap as yaml file
kubectl delete -f [fileName.yaml]

Secrets

Secret Type

Secret Type

Secrets: stored as base64 encoded string (not secure)

Probes

Probes: for a Pod to know the status of a application inside container

Probes are not component, but used by Kubelet.

Horizontal Pod Autoscaling

HorizontalPodAutoscaling Commands

kubectl get hpa [horizontalPodAutoscalingName]
kubectl delete hpa [horizontalPodAutoscalingName]
kubectl delete -f [fileName.yaml]

To use it, you need to install metrics-server (and add --kubelet-insdecure-tls if on DockerDesktop)

Kubernetes in Home Cluster

Firstly, make sure you have no swap

sudo swapoff -a
sudo nano /etc/fstab # and remove swap

Then install kubeadm kubelet kubectl kubernetes-cni after you set up repository following this guide on both worker nodes and the master node.

Then run sudo kubeadm init only on master node, then join the worker nodes with the command output.

Then, copy configuration file to $HOME/.kube/config on the master node (see Installation)

If you have problems [ERROR CRI]: container runtime is not running, then run the following

sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.bak
sudo systemctl restart containerd
sudo kubeadm init

In the end, you will get

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 104.171.203.218:6443 --token lzwo36.sjbuvacn3aube42l \
        --discovery-token-ca-cert-hash sha256:b3b3db068180a8a7edf1d3025e77734974af735b421de4396ddd131ce61f4a65

Use the kubeadm join commands on the worker nodes.

Then you can see

❯ kubectl get nodes
NAME              STATUS     ROLES           AGE    VERSION
104-171-203-218   NotReady   control-plane   5m6s   v1.27.4

We need to install Calico for its network security solution on master node:

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

Then wait for a while, you should see the nodes are "Ready"

Nvidia CUDA GPU

It is kinda difficult to enable GPU on kubernetes for couple reasons:

  1. the image with driver in it is huge (you may find docker system prune -a helpful, tar-ing file with pigz might speedup work but I never get the format to work)
  2. there are many deprecated methods online that you don't know whether they work (such as nvidia-docker, nvidia-docker2 at Here or pytorch-operator)
  3. different node vendor has different things (AMD, Intel, NVIDIA all provide k8s support)
  4. image need to be compatible with node driver
  5. things may depend on nvcc to compile
  6. may not be simulator-agnostic
  7. According to Here, you must use docker engine instead of containerd engine to enable GPU.

Follow Docker

Before proceed to next sections, you need to make sure your Dockerfile is configured correctly. That means docker run -it --entrypoint /bin/bash [image-hash] should allow you to access nvidia-smi and pytorch should acknowledge the GPU. Note that you must use image hash for local images. (Also local images should not be tagged latest, otherwise it will always be pulled from dockerhub)

You can check GPU avaliability by:

import torch
torch.cuda.is_available()

To remove docker container, do docker ps -a, then docker remove [containerId]

Follow k8s-device-plugin

Follow k8s-device-plugin instructions.

Make sure which nvidia-container-toolkit and which nvidia-container-runtime exist (you might want to reinstall nvidia-container-toolkit) no matter which solution you are following.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/libnvidia-container.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

Edit /etc/docker/daemon.json to be: and then sudo systemctl restart docker (or you may choose to run sudo nvidia-ctk runtime configure instead as it might auto configure this file for you?)

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

Edit /etc/containerd/config.toml to be: and then sudo systemctl restart containerd

Also do sudo systemctl daemon-reload

version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "nvidia"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime"

You don't need to do kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.1/nvidia-device-plugin.yml, as we choose to follow microk8s' setup. This is because following k8s-device-plugin does not work fully for me.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.14.1
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

If you do follow above and do a kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.1/nvidia-device-plugin.yml (it will create a deamonset in a different namespace kubectl get namespaces) and look at its log, you will get [factory.go:115] Incompatible platform detected. If you ignore this problem and try to schedule a GPU job, you will get 0/1 nodes are available: 1 Insufficient nvidia.com/gpu

Follow Microk8s

Use microk8s inspect to check microk8s installation.

When you setup everything above, reboot the system, then do microk8s enable gpu, you will get:

Infer repository core for addon gpu
Enabling NVIDIA GPU
Addon core/dns is already enabled
Addon core/helm3 is already enabled
Checking if NVIDIA driver is already installed
GPU 0: NVIDIA RTX A2000 8GB Laptop GPU (UUID: GPU-11886664-6f9b-bb01-311a-573dffe8aee4)
Using host GPU driver
"nvidia" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nvidia" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: gpu-operator
LAST DEPLOYED: Mon Sep  4 06:29:55 2023
NAMESPACE: gpu-operator-resources
STATUS: deployed
REVISION: 1
TEST SUITE: None
NVIDIA is enabled

The command will also add many pods. You should wait for a long time (like 5min), then do microk8s kubectl logs -n gpu-operator-resources -lapp=nvidia-operator-validator -c nvidia-operator-validator to verify successful configuration. You should get all validations are successful.

After running klzzwxh:0064

After running microk8s enable gpu

Now, when you do kubectl describe node -A | grep nvidia, you should see:

                    nvidia.com/cuda.driver.major=520
                    nvidia.com/cuda.driver.minor=61
                    nvidia.com/cuda.driver.rev=05
                    nvidia.com/cuda.runtime.major=11
                    nvidia.com/cuda.runtime.minor=8
                    nvidia.com/gfd.timestamp=1693823531
                    nvidia.com/gpu.compute.major=8
                    nvidia.com/gpu.compute.minor=6
                    nvidia.com/gpu.count=1
                    nvidia.com/gpu.deploy.container-toolkit=true
                    nvidia.com/gpu.deploy.dcgm=true
                    nvidia.com/gpu.deploy.dcgm-exporter=true
                    nvidia.com/gpu.deploy.device-plugin=true
                    nvidia.com/gpu.deploy.driver=true
                    nvidia.com/gpu.deploy.gpu-feature-discovery=true
                    nvidia.com/gpu.deploy.node-status-exporter=true
                    nvidia.com/gpu.deploy.operator-validator=true
                    nvidia.com/gpu.family=ampere
                    nvidia.com/gpu.machine=21D6004WUS
                    nvidia.com/gpu.memory=8192
                    nvidia.com/gpu.present=true
                    nvidia.com/gpu.product=NVIDIA-RTX-A2000-8GB-Laptop-GPU
                    nvidia.com/gpu.replicas=1
                    nvidia.com/mig.capable=false
                    nvidia.com/mig.strategy=single
  nvidia.com/gpu:     1
  nvidia.com/gpu:     1
  gpu-operator-resources      nvidia-container-toolkit-daemonset-kjp55                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
  gpu-operator-resources      nvidia-dcgm-exporter-kd852                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
  gpu-operator-resources      nvidia-device-plugin-daemonset-fq2rk                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
  gpu-operator-resources      nvidia-operator-validator-q9hz9                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
  nvidia.com/gpu     0           0

Summary

#!/bin/sh

# nvidia-container-toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/libnvidia-container.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# microk8s GPU
microk8s inspect
microk8s enable gpu
sudo nvidia-ctk runtime configure
sudo systemctl restart containerd
sudo systemctl restart docker
sudo systemctl daemon-reload
microk8s kubectl logs -n gpu-operator-resources -lapp=nvidia-operator-validator -c nvidia-operator-validator

# k9s
mkdir k9s
cd k9s
wget https://github.com/derailed/k9s/releases/download/v0.27.4/k9s_Linux_amd64.tar.gz
tar -xzf k9s_Linux_amd64.tar.gz
sudo cp k9s /usr/local/bin/k9s
cd ..
rm -rf k9s

Other Microk8s Instructions

For production and more advanced setup, follow Here

You might get the following error Normal Scheduled 2m21s default-scheduler Successfully assigned ***-77c979f8bf-px4v9 to gke-***-389a7e33-t1hl Warning Failed 20s kubelet Error: context deadline exceeded Normal Pulled 7s (x3 over 2m20s) kubelet Container image "***" already present on machine Warning Failed 7s (x2 over 19s) kubelet Error: failed to reserve container name ***-77c979f8bf-px4v9_***": name "***-77c979f8bf-px4v9_***" is reserved for "818fcfef09165d91ac8c86ed88714bb159a8358c3eca473ec07611a51d72b140"

See Here

To deal with this issue, you might want to increase load time by appending --runtime-request-timeout 30m0s (See all settings Here and instructions Here) to the file /var/snap/microk8s/current/args/kubelet. Then restart microk8s stop, microk8s start

To build and use images locally, you might do the following (don't use :latest tag):

docker save frontend:3.4 > tmp-image.tar
microk8s ctr image import tmp-image.tar

If you find your image in microk8s ctr images ls, then microk8s can load your image.

However, docker save and docker load is often threshold by io, so we build a local registry: But local registry is faster to build, but slower for ContainerCreating. So in development, don't use local registry.

# we need to enable this local registry function before hand
microk8s enable registry:size=20Gi

# to directly build image into local registry
docker build . -t localhost:32000/image_name:image_tag
# or import from existing image
docker tag some_hash_tag localhost:32000/image_name:image_tag

# finally push to registry
docker push localhost:32000/image_name

And then use image: localhost:32000/image_name:image_tag in kubernetes. To check built image, checkout curl http://localhost:32000/v2/_catalog

Pushing to this insecure registry may fail in some versions of Docker unless the daemon is explicitly configured to trust this registry. To address this we need to edit /etc/docker/daemon.json and add, then do sudo systemctl restart docker: { "insecure-registries" : ["localhost:32000"] } (also if docker failed to restart, you might check out dockerd for error messages)

See Here for details.

Table of Content