Provision VMs on KVM with Terraform

Some months ago I attended to a talk in which it was explained how to provision infrastructure in AWS with Terraform, I was talking with @raularanda (the host of that talk) about the idea of making a lab similar but with KVM, this entry it’s the result of that conversation.

Terraform is a software which can help the user manage, build and change cloud infrastructure using code, it can interact with various API’s using providers (plugins), these peaces of software are a sort of intermediary between Terraform and those API’s. Terraform can be used to manage the Lifecycle of various created ‘resources’ of those API’s. A resource can be any kind of type of virtual device but also be just a file or a directory.

KVM (Kernel-based Virtual Machine) is an open-source virtualization system for Linux systems. It’s a hypervisor that can be used to create virtualized, hardware accelerated virtual machines. It can achieve this using the built-in QEMU hardware emulator to emulate processors. QEMU and KVM are both configured using XML. In general, KVM/QEMU are used together in combination with a virtualization library called libvirt, but they can be used without it.

Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations

So to provision a VM in KVM with Terraform is needed to describe some computing resources to support the VM and configure it using Terraform-Libvirt-provider plugin, the resources could be:

  • CPUs (vpcu’s)
  • Memory
  • Users (SSH keys)

Setup libvirtd

In this article is assumed that you have installed KVM, it’s not your case you could check this entry, but for this lab it’s needed to turn off SElinux security driver for libvirt (enabled by default) because it will prevent from creating a domain:

# vim /etc/libvirt/qemu.conf
...
security_driver= [ "none" ]
# systemctl restart libvirtd
# systemctl status libvirtd
...
     Active: active (running) since Sun 2020-02-02 17:40:18 CET; 13min ago
...
libvirtd[149803]: Configured security driver "none" disables default policy to create confined guests

Libvirt requires the host machine’s user permissions to be able to allocate computing resources. 

# usermod -aG kvm,libvirt <USER>
$ su - <USER>
$ id -nG
<USER_GRP> ... kvm ... libvirt

Setup a workspace

To have all our work organized it’s strongly recommended to create a directory structure to keep all needed files in our project, images and downloads, in one place:

$ mkdir -p /mnt/libvirt/terraform/images/debian_pool /mnt/libvirt/terraform/downloads

Download image needed to this project, as we are going to base our VM on Debian stable we need to download it in our downloads directory:

$ cd /mnt/libvirt/terraform/downloads
$ wget https://cdimage.debian.org/debian-cd/10.2.0/amd64/iso-cd/debian-10.2.0-amd64-netinst.iso

Installing Terraform

# wget https://releases.hashicorp.com/terraform/0.12.20/terraform_0.12.20_linux_amd64.zip
$ unzip terraform_0.12.20_linux_amd64.zip
# mv terraform /usr/local/bin/
$ terraform -v
Terraform v0.12.20
+ provider.libvirt (unversioned)
+ provider.template v2.1.2

# mkdir /mnt/libvirt/terraform
# chown <USER>:<USER> /mnt/libvirt/terraform
$ cd /mnt/libvirt/terraform && terraform init
Terraform initialized in an empty directory!

Installing Libvirt provider

$ cd ~/.terraform.d
$ mkdir plugins
$ wget https://github.com/dmacvicar/terraform-provider-libvirt/releases/download/v0.6.0/terraform-provider-libvirt-0.6.0+git.1569597268.1c8597df.Ubuntu_18.04.amd64.tar.gz
$ tar xvf terraform-provider-libvirt-0.6.0+git.1569597268.1c8597df.Ubuntu_18.04.amd64.tar.gz
$ mv terraform-provider-libvirt ~/.terraform.d/plugins/

Setup VM disk

At this point we’re going to clone our already semi-configured image created following this approach, as a requirement for this step a VG vg-kvm should be created and also a LV called master, as described in .

# lvs
  LV                   VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
...
  master                 vg-kvm  -wi-a-----  10.00g                                                    

# qemu-img convert -f raw -O qcow2 /dev/vg-kvm/master /mnt/libvirt/terraform/images/debian_pool/base.qcow2
# chown <USER>:<USER> /mnt/libvirt/terraform/images/debian_pool/base.qcow2
$ qemu-img info /mnt/libvirt/terraform/images/debian_pool/base.qcow2
image: /mnt/libvirt/terraform/images/debian_pool/base.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.48 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
$ ls -lh /mnt/libvirt/terraform/images/debian_pool/base.qcow2
-rw-r--r-- 1 <USER> <USER> 2.5G Feb  2 17:30 /mnt/libvirt/terraform/images/debian_pool/base.qcow2

Setup Terraform configuration (Libvirt Provider)

Let’s create libvirt.tf file for your VM deployment on KVM, Terraform uses ‘.tf’ files to define its configuration (using Hashicorp scripting language). Our target is to create a new VM and allocate the resources needed to support it. In ‘.tf’ files it can be found ${} to call functions and # to define comments. So let’s create a pool, volume, cloud_init config, and a domain:

  • Resource definition: resource "resource_type" "resource_name" { ... }
    • Resource type: resource we want to provide for a certain provider, libvirt_pool: the store to be used by VM images
    • Resource name: user resource identifier (debian)
      • Name: user pool identifier (debian-pool)
      • Type: type dir provides the means to manage files within a directory, files here could have formats like qcow, vmdk or just raw
      • Path: storage location
# add the provider, this code will connect to Hypervisor using libvirt
provider "libvirt" {
  uri = "qemu:///system"
}

# create pool
resource "libvirt_pool" "debian" {
 name = "debian-pool"
 type = "dir"
 path = "${path.module}/images/debian_pool"
}

# create image volume
resource "libvirt_volume" "image-raw" {
 name = "debian-buster-amd64.qcow2"
 ##name = ${var.name}
 pool = libvirt_pool.debian.name
 source = "${path.module}/images/debian_pool/base.qcow2"
 ##source = "https://cdimage.debian.org/cdimage/openstack/current-10/debian-10-openstack-amd64.qcow2"
 #format = "raw"
 format = "qcow2"
}

# add cloudinit disk to pool
resource "libvirt_cloudinit_disk" "commoninit" {
 name = "commoninit.iso"
 pool = libvirt_pool.debian.name
 user_data = data.template_file.user_data.rendered
}

# read the configuration
data "template_file" "user_data" {
 template = file("${path.module}/cloud_init.cfg")
}

# Define KVM domain to create
resource "libvirt_domain" "master-k8s" {
  name   = "master-k8s"
  memory = "4096"
  vcpu   = 2

  network_interface {
    network_name = "default"
  }

  disk {
    volume_id = libvirt_volume.image-raw.id
  }

  console {
    type = "pty"
    target_type = "serial"
    target_port = "0"
  }

  # type = "none" not allowed by the provider
  graphics {
    type = "spice"
    listen_type = "address"
    autoport = true
  }
}

Setting up cloud-init for configuring the VM

Create a file called cloud_init.cfg which is going to use yaml syntax to create VM users

$ vim /mnt/libvirt/terrraform/cloud_init.cfg
#cloud-config
users:
 - name: foo
 sudo: ALL=(ALL) NOPASSWD:ALL
 groups: users, admin
 home: /home/foo
 shell: /bin/bash
 ssh_authorized_keys:
 - ssh-rsa <PUB_KEY>
# install packages
packages:
 - git

<PUB_KEY> has to be replaced by the content of this file ~/.ssh/id_rsa.pub

/mnt/libvirt/terraform] $ terraform init
Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
...
* provider.template: version = "~> 2.1"
Terraform has been successfully initialized!
...
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform providers
.
├── provider.libvirt
└── provider.template
$ terraform plan
$ terraform apply

$ virsh pool-list
 Name          State    Autostart
-----------------------------------
 debian-pool   active   yes
$ virsh list --all
 Id   Name            State
--------------------------------
 1    master-k8s      running
$ virsh console master-k8s
##Ctrl+ 5 to exit virsh console

$ terraform destroy
...
Destroy complete! Resources: 4 destroyed.

Reference links:

Happy Terraforming!


“There is only one corner of the universe you can be certain of improving, and that’s your own self.”
–Aldous Huxley

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s