In one of my previous articles I described how to install and setup a basic K8s cluster on Debian GNU/Linux, regarding network configuration I followed a very standard configuration attaching ethernet NIC to a bridge, but recently I required to connect my VMs just using my wireless card, and I would like to explain the process how I setup it up.
Previous concepts: bridging and routing
The main difference between these two concepts are that routing operates on Layer 3 (IP packets level) while bridging operates on Layer 2 (MAC level, Ethernet frames), which means that if we choose routing option we would have 2 separate networks, which can talk with each other.
But there are 2 different LANs, with different IP networks. On the other hand bridging makes them a unified LAN, with all IP addresses on the same network.
First approach: NAT-based network
Let’s keep the first approach as the simplest one because the only thing we need is an outbound IPv4 network access, in this case libvirtd act as a router. I have my wifi card, wlp3s0, to connect to Internet.
$ virsh net-dumpxml default <network> <name>default</name> ... <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='br0' stp='off' delay='0'/> <mac address='6e:7b:73:a6:a3:4c'/> <ip address='192.168.1.2' netmask='255.255.255.0'>...</ip> </network> $ ip a show br0 6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 6e:7b:73:a6:a3:4c brd ff:ff:ff:ff:ff:ff inet 192.168.1.2/24 brd 192.168.1.255 scope global br0 valid_lft forever preferred_lft forever $ ip a show br0-nic 7: br0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master br0 state DOWN group default qlen 1000 link/ether 6e:7b:73:a6:a3:4c brd ff:ff:ff:ff:ff:ff $ ip r default via 10.111.11.1 dev wlp3s0 proto dhcp metric 600 10.111.11.0/24 dev wlp3s0 proto kernel scope link src 10.111.11.112 metric 600 192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.2
Guest VM configuration is pretty simple, in my case I choose an assign statically its own IP:
$ virsh edit node1 ... <interface type='network'> ... <source network='default'/> <model type='virtio'/> ... </interface> $ virsh start node1 $ virsh domiflist node1 Interface Type Source Model MAC ------------------------------------------------------------- vnet0 network default virtio 52:54:00:d8:48:6e $ ip a show vnet0 15: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000 link/ether fe:54:00:d8:48:6e brd ff:ff:ff:ff:ff:ff $ ssh node1 node1:~$ ip route get 126.96.36.199 188.8.131.52 via 192.168.1.2 dev enp1s0 src 192.168.1.4 uid 1000 cache node1:~$ ip r get 192.168.1.2 192.168.1.2 dev enp1s0 src 192.168.1.4 uid 1000 cache
It works for me. If you want to use your own NAT-based network you could follow these instructions.
You also could use a routed network as it’s explained here.
First approach drawbacks
- Libvirtd automatically inserts iptables rules, check this out.
Second approach: Create a Virtual Guest network which is a Host LAN subnet
In the second approach we are going to create a new network which shares the same /24 space with your network in our case we have our LAN defined as 10.111.11.0/24 so we can use to this experiment to 10.111.11.128/28, the only thing that we need to take care it would be that this new network can’t share the same IPs so let’s check my current DHCP configuration to confirm the IP range 10.111.11.112 – 10.111.11.127. To create the file to define the new network you should know:
- Wireless device name: wlp3s0
- Network: 10.111.11.128/28
- Netmask: 255.255.255.240
- Broadcast: 10.111.11.143
- Host Min: 10.111.11.129
- Host Max: 10.111.11.142
- Host: 14
- DHCP Range: 10.111.11.136 – 10.111.11.142
$ cat proxyArp.xml <network> <name>proxyArp</name> <forward dev="wlp3s0" mode="route"> <interface dev="wlp3s0"/> </forward> <bridge name="virbr0" stp="on" delay="0"/> <mac address="52:54:00:69:ab:d6"/> <domain name="proxyArp"/> <ip address="10.111.11.129" netmask="255.255.255.240"> <dhcp> <range start="10.111.11.136" end="10.111.11.142"/> </dhcp> </ip> </network> $ virsh net-create --file proxyArp.xml Network proxyArp created from proxyArp.xml $ virsh net-autostart proxyArp $ virsh net-start proxyArp $ virsh net-list --all Name State Autostart Persistent --------------------------------------------- default active yes yes proxyArp active yes yes $ ip a ... 14: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether b2:c2:eb:b7:f1:05 brd ff:ff:ff:ff:ff:ff inet 10.111.11.129/28 brd 10.111.11.143 scope global virbr0 valid_lft forever preferred_lft forever 15: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:69:ab:d6 brd ff:ff:ff:ff:ff:ff $ ip r default via 10.111.11.1 dev wlp3s0 proto dhcp metric 600 10.111.11.0/24 dev wlp3s0 proto kernel scope link src 10.111.11.112 metric 600 10.111.11.128/28 dev virbr0 proto kernel scope link src 10.111.11.129 linkdown
Creating a new VM you could setup manually your network configuration in this way (tested with Debian installation process):
Static IP: 10.111.11.136 Netmask: 255.255.255.240 Gateway: 10.111.11.129 DNS Server: 10.111.11.1
If we have existing VMs it’s needed to change its configuration to use this new network:
$ virsh list --all Id Name State ----------------------------- - k8s-master shut off - node1 shut off - node2 shut off $ virsh edit k8s-master ... <interface type='bridge'> ... <source bridge='virbr0'/> </interface> ## The same modification to all your VMs $ virsh edit node1 ...
Configuring proxy ARP
To manage the routing issues generated with this configuration which are essentially that if a host (connected to our host network 10.111.11.112 – .127) wants to send an IP packet to the Guest (from .136 to .142). That host knows that it’s in the same LAN, so it has to find out what is its MAC Address (Layer 2). To find this out, hosts sends an ARP (Address Resolution Protocol) broadcast packet, asking “which is the MAC address assigned to .136 IP address”. ARP dictates that only the owner of that IP address should response, so Virtual Host will ignore the packet. Furthermore, it wouldn’t route it to the Virtual Guest, because no one told him to route Broadcast packets – it only routes IP packets send specifically to .136.
# arp -v Address HWtype HWaddress Flags Mask Iface master.acme.local (incomplete) br0 XXXXXXXXXXXXX ether b8:27:eb:7c:38:ed C wlp3s0 10.111.11.126 ether 40:a5:ef:bb:b4:80 C wlp3s0 ... # arp -i wlp3s0 -Ds 10.111.11.136 wlp3s0 pub
And this is working for me! Once you make your tests you could delete this network using these commands:
$ virsh net-destroy proxyArp Network proxyArp destroyed $ ip a|grep vir-br0 $ virsh net-list --all Name State Autostart Persistent ----------------------------------------------- default active yes yes proxyArp inactive yes yes $ virsh net-undefine proxyArp Network proxyArp has been undefined $ virsh net-list --all Name State Autostart Persistent -------------------------------------------- default active yes yes
Second approach drawbacks
- Proxy ARP command is just to one IP not for the entire subnet
- ARP table is cleaned every reboot so it’s needed to setup a startup script
- If you change your Access Point and connected network changes you should modify settings in your Virtual Guest configuration
Third approach: using TAP devices
# sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward" # ip tuntap add mode tap tap0 user <USER> # ip addr add 10.10.10.10/24 dev tap0 # ip link set tap0 up # apt install parprouted # parprouted wlp3s0 tap0 ##Adding some routing tables entries to allows packets to travel through the ends of the tap device: # iptables-save > iptables_`date +%Y%m%d` # iptables -A INPUT -i tap0 -j ACCEPT # iptables -A FORWARD -i tap0 -j ACCEPT # iptables -A FORWARD -o tap0 -j ACCEPT
In the guest we only would need to setup statically our LAN configuration:
# ip addr add 10.111.1.10/24 dev eth0 # vim /etc/network/interfaces auto eth0 iface eth0 inet static address 10.111.1.10 netmask 255.255.255.0 network 10.111.1.0 broadcast 10.111.1.255 gateway 10.111.1.1
Additionally is needed to change interface defined in your already created VMs:
$ virsh edit k8s-master ... <interface type='network'> <source network='default'/> <target dev='tap0'/> <model type='virtio'/> ... </interface>
If we are happy with this configuration you should keep persistent by adding the following settings to /etc/network/interfaces (notice manual type instead of static to be able to modify routing table):
# vim /etc/network/interfaces iface tap0 inet manual pre-up ip tuntap add tap0 mode tap user root pre-up ip addr add 10.111.1.10/24 dev tap0 up ip link set dev tap0 up post-up ip route del 10.111.1.0/24 dev tap0 post-up ip route add 10.111.1.10/32 dev tap0 post-up iptables -A INPUT -i tap0 -j ACCEPT post-up iptables -A FORWARD -i tap0 -j ACCEPT post-up iptables -A FORWARD -o tap0 -j ACCEPT post-down ip link del dev tap0
Third approach drawbacks
- With this method you can’t use DHCP in your Guests
- We need a tap interface by VM
Optional: Check if your Wireless NIC supports bridging
Previous approachs could be avoid if our wireless card supports bridging so you could get some information about the chipset and the driver used to know if you have implemented this feature.
# update-pciids $ lspci | grep -i intel|grep -i wireless 03:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78) # lspci -vv -s 03:00.0 03:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78) Subsystem: Intel Corporation Dual Band Wireless-AC 8265 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 142 Region 0: Memory at f2200000 (64-bit, non-prefetchable) [size=8K] ... Kernel driver in use: iwlwifi $ lspci -nn -s 03:00.0 03:00.0 Network controller : Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78) $ apt install vlan && modinfo iwlwifi|less $ lshw -C network | grep -B 1 -A 12 'Wireless i' *-network DISABLED description: Wireless interface product: Wireless 8265 / 8275 vendor: Intel Corporation physical id: 0 bus info: pci@0000:03:00.0 logical name: wlp3s0 version: 78 serial: ee:74:17:11:08:92 width: 64 bits clock: 33MHz capabilities: bus_master cap_list ethernet physical wireless configuration: broadcast=yes driver=iwlwifi driverversion=5.4.0-2-amd64 firmware=22.361476.0 latency=0 link=no multicast=yes wireless=IEEE 802.11 resources: irq:142 memory:f2200000-f2201fff
But man is not made for defeat. A man can be destroyed but not defeated.
— Ernest Hemingway