Installing a K8s cluster on Debian Bullseye with KVM

Five years ago I wrote this blog entry talking about KVM, at that time I needed to run Microsft Windows 7 VM to make some tests for my work.

This time I need to make a lab with a couple of VMs to run a Kubernetes cluster in my laptop and I’ve decided to use KVM to create the required infrastructure.

Checking Requirements

First of all let’s check if we have enable virtualization at BIOS level we can check it running:

# dmesg |grep -i kvm
[    3.614335] kvm: disabled by bios

As it’s shown in log virtualization needs to be enabled entering in BIOS menu to enable VT-X options, with this change in the next boot it can be found in boot log:

# dmesg |grep -i virt
[    1.017892] DMAR: Intel(R) Virtualization Technology for Directed I/O

In addition is also required to check if our CPU support hardware virtualization, we are going to include encryption check running this command:

$ egrep -wo 'vmx|lm|aes' /proc/cpuinfo | sort | uniq\
| sed -e 's/aes/Hardware encryption=Yes (&)/g' \
-e 's/lm/64 bit cpu=Yes (&)/g' -e 's/vmx/Intel hardware virtualization=Yes (&)/g'
Hardware encryption=Yes (aes)
64 bit cpu=Yes (lm)
Intel hardware virtualization=Yes (vmx)

Reviewing information from our CPU from /proc/cpuinfo we can find these flags:
lm – this flag means your system has a 64 bit CPU (Intel or AMD)
vmx – Intel VT-x, virtualization support
aes – AES/AES-NI advanced encryption support

Installing software

Let’s install all needed packages:

# apt install qemu-kvm libvirt-clients libvirt-daemon-system virtinst bridge-utils

virtinst package would be required if you want to use virt-install command.

To connect to local libvirt:

# adduser  libvirt
$ virsh list --all
Id   Name   State
--------------------

We are going to use libvirt which is an open-source API, daemon and management tool for managing platform virtualization to make easier manager our KVM environment.

Setting up our network (host)

As a requirement we need that our VMs are connected between them because they are going be part of a cluster so let’s setup a bridged networking :

# ip link add name br0 type bridge
# ip link set dev br0 up
# ip link set dev enp0s31f6 master br0
$ ip a s
...
5: br0:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6e:7b:73:a6:a3:4c brd ff:ff:ff:ff:ff:ff

In my case enp0s31f6 is the host ethernet interface to be connected to the bridge you can find yours using this command:

$ ip -f inet address show
# Or in a short way
$ ip -f inet a s

We’re setting up our bridge using a separated file located here /etc/network/interfaces.d/br0:

iface br0 inet static
	address 192.168.1.2
	broadcast 192.168.1.255
	netmask 255.255.255.0
	gateway 192.168.1.1
	bridge_ports enp0s31f6
	bridge_stp off       # disable Spanning Tree Protocol        
        bridge_waitport 0    # no delay before a port becomes available        
        bridge_fd 0          # no forwarding delay

Let’s check our new configuration restarting network-manager service (you should check if your network set up is controlled by Network Manager):

# systemctl restart network-manager
# systemctl status network-manager
$ ip a s br0
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6e:7b:73:a6:a3:4c brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever

Edit /etc/sysctl.conf file adding the following options:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.ipv4.ip_forward = 1

A reboot is required, although let’s check bridge setup within KVM:

$ virsh net-list --all
Name      State      Autostart   Persistent
----------------------------------------------
default   inactive   no          yes

To get br0 MAC address run:

$ ip address show dev br0 | awk '$1=="link/ether" {print $2}'

If we want to modify some information to setup the range in DHCP it can be done like that:

$ export LIBVIRT_DEFAULT_URI='qemu:///system'
$ virsh net-edit default ...
$ virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>XXXXXXXX</uuid>
  <forward mode='nat'/>
  <bridge name='br0' stp='off' delay='0'/>
  <mac address='6e:7b:73:a6:a3:4c'/>
  <ip address='192.168.1.2' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.1.3' end='192.168.1.8'/>
    </dhcp>
  </ip>
</network>
$ virsh net-start default
$ virsh net-autostart default
Network default started
$ virsh net-list --all
Name      State    Autostart   Persistent
--------------------------------------------
default   active   yes         yes

Managing VM’s from command line

Some of the most commands to work with VMs:

$ export LIBVIRT_DEFAULT_URI='qemu:///system'
$ virsh start k8s-master #Boot a VM
$ virsh shutdown k8s-master #Stop a VM
$ virsh suspend k8s-master
## Delete a VM (destroy and undefine):
$ virsh destroy k8s-master
$ virsh undefine k8s-master
Domain k8s-master has been undefined
### Remember to remove image disk file to release space in your host

With our bridge correctly configured we can setup next VM deployments using this flag:

$ virt-install --network bridge=br0...

Let’s script VM creation

Let’s define a script to create our VM’s (memory, disk and vcpu will depend on your host hardware):

#!/bin/sh
if [ -z "$1" ] ;
then
 echo Specify a VM name.
 exit 1
fi
virt-install \
--name $1 \
--ram 4096 \
--disk path=/your/path/images/$1.img,size=10 \
--vcpus 2 \
--os-type linux \
--os-variant debian10 \
--network bridge:br0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--location /your/path/images/debian-10.1.0-amd64-netinst.iso \
--extra-args 'console=ttyS0,115200n8 serial'

Let’s create our first VM:

$ create_vm.sh k8s-master

In this tutorial I’m not going to describe how to setup a Debian GNU/Linux VM but some minimal requirements in our case could be:

• NOT setup Swap partition (Manual partition option to just create one partition)
• If you have swap enabled you could disable swap using these commands:
  • # swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab
• Install OpenSSH server and Standard packages (disable Desktop environment and Print server)
• Setup static IP configuration

Configuring our master node (control plane)

Once we have our VMs setup it’s time to install kubernetes cluster software steps in MASTER NODE:

# apt install -y docker.io curl sudo gnugpg
# cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
# mkdir -p /etc/systemd/system/docker.service.d
# systemctl daemon-reload
# systemctl restart docker
$ curl -sL https://gist.githubusercontent.com/neklaf/92ff1b15b26c8bd42feb5755586d1af5/raw/c786c40959303e7d51485c271535cbb9bdd6a638/config.sh | sudo sh
# curl https://docs.projectcalico.org/v3.9/manifests/calico.yaml -O 
# vim calico.yaml
...
- name: CALICO_IPV4POOL_CIDR
value: "192.168.0.0/24"
# kubeadm init --pod-network-cidr 192.168.0.0/24
...
Your Kubernetes control-plane has initialized successfully!
...
# mkdir -p $HOME/.kube
# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# chown $(id -u):$(id -g) $HOME/.kube/config
# kubectl apply -f calico.yaml
# kubectl get nodes
 NAME     STATUS   ROLES    AGE     VERSION
 master   Ready    master   7m58s   v1.15.0
# source <(kubectl completion bash)
# echo "source <(kubectl completion bash)" >> ~/.bashrc
(OPTIONAL) # kubectl taint nodes --all node-role.kubernetes.io/master-
 node/master untainted

Let’s create a Discovery Token CA Hash in the master node to ensure that a worker joins the cluster in a secure way. To generate this hash:

# openssl x509 -pubkey \
   -in /etc/kubernetes/pki/ca.crt |openssl rsa \
   -pubin -outform der 2>/dev/null |openssl dgst \
   -sha256 -hex |sed 's/^.* //'
 06dfd9ea6a48fe5a985f7041dd28ae567d797b6140cab2e10360ebe95f0bd58 

This directory keeps all manifest files: /etc/kubernetes/manifests

Setting up a worker node

Steps in a WORKER NODE:

# apt install -y docker.io curl sudo gpg
# cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
# mkdir -p /etc/systemd/system/docker.service.d
# systemctl daemon-reload
# systemctl restart docker
$ curl -sL https://gist.githubusercontent.com/neklaf/92ff1b15b26c8bd42feb5755586d1af5/raw/412c31c364fd0cc87e49fac530d150dc6bcf2a06/config.sh | sudo sh
# kubeadm join --token qziqar.oou2omii1bdwi64e <MASTER-IP>:6443 --discovery-token-ca-cert-hash sha256:06dfd9ea6a48fe5a985f7041dd28ae567d797b6140cab2e10360ebe95f0bd58b
...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
master# scp -r ~/.kube root@<WORKER-IP>:

If your token has expired you should follow these steps:

master# kubeadm token create
gmovhu.m6dw6x3meehusbr1
master# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
gmovhu.m6dw6x3meehusbr1   23h       2020-01-29T18:56:56+01:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
node2:~# kubeadm join --token gmovhu.m6dw6x3meehusbr1 master:6443 --discovery-token-ca-cert-hash sha256:06dfd9ea6a48fe5a985f7041dd28ae567d797b6140cab2e10360ebe95f0bd58b
master# scp ~/.kube/config node2:./kube/config
node2:~# kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   64d     v1.15.0
node1    Ready    <none>   64d     v1.15.0
node2    Ready    <none>   3m17s   v1.15.0

Let’s check in the master node if we can see the new node added to the cluster (has to be checked in all cluster nodes):

master# kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   58m     v1.15.0
node1    Ready    <none>   2m17s   v1.15.0
master# kubectl run nginx --image nginx
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/nginx created
worker# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
nginx-7bb7cd8db5-ghmwv   1/1     Running   0          40m   192.168.0.129   node1   <none>           <none>

Both master and worker should have these network interfaces you run this command to make a quick check:

# ip a| grep -e dock -e cali -e tun
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.0.128/32 brd 192.168.0.128 scope global tunl0
7: cali9ee1dd76ccc@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default

Note that if something was wrong during installation process probably the quickest way to fix the problem could be to restart a fresh installation process quickly destroying VM and starting again.

References:

• https://blog.alexellis.io/kvm-kubernetes-primer/
• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
• https://jamielinux.com/docs/libvirt-networking-handbook/bridged-network.html
• https://wiki.libvirt.org/page/Networking

Enjoy your new fresh k8s cluster!
—
First they ignore you,
then they laugh of you,
then they try to copy you.
Then you change the world.
— Elisabeth Holmes