K22: Virtual Machine (VM) Setup Instructions

For kernel development and experimentation in this class, you will be working with Linux inside a controlled environment on your own machine using a Linux VM.

This tutorial explains how to get a recent Ubuntu distribution and run a VM with it, as well as how to obtain, compile, install, and run a recent Linux kernel inside that VM. Finally, we will create and run our first kernel patch!

The steps detailed in this tutorial are a strong recommendation from us, based on what we used on our own machines and found to work reliably. Feel free to use and setup any VM of your liking, as long as (1) you can make it boot with a recent Ubuntu distribution, and most importantly (2) be able to obtain, edit, patch, compile, install, and run a patched (modified) kernel.

Option-1: Set up a VM with VMware (Windows and Ubuntu users)

In what follows in this section, we will be describing steps regarding using VMware Workstation as the hypervisor, and assume that you are running it on a Linux-based system.

Download and Install VMware

1) Create an account at Broadcom.com. (Your username is your email!)

2) Log in to Broadcom Support Portal.

3) To use the free edition of VMware Workstation Pro, go to Download VMware Workstation Pro (Free) page and download the latest release. (You have to download and save the .bundle file on your computer.)

On your local machine

4) Navigate the folder where the VMware Workstation bundle installer is located. (By default, it is at the Downloads directory.)

5) Make the file executable and run the corresponding installer for your system.

chmod +x VMware-Workstation-Full-####-####.architecture.bundle
sudo ./VMware-Workstation-Full-####-####.architecture.bundle

6) Follow the instructions displayed on the screen to complete the installation.

7) After the installation finishes, reboot your machine.

Download an Ubuntu ISO Image

We will work with Ubuntu 26.04.1 LTS.

Enable Hardware Virtualization

Visit your motherboard manufacturer’s website (or your laptop’s manufacturer’s website if you are using a laptop) and find the page for your specific device. In the manuals of the manufacturer you will find the way to enter the UEFI/BIOS menu settings of your device and edit them. Look for settings with names like Virtualization Support, VMX, Enable virtualization. It should be a boolean toggle (true/false or enabled/disabled) which you want to ensure is turned on (enabled, true).

Create a VM

Now we are ready to create a Linux VM.

1) Run VMware at the background with the command: /usr/bin/vmware &.

2) To create a new VM using the Ubuntu ISO image that you downloaded, go to the top-left corner, click File, and then select New Virtual Machine.

3) In the Virtual Machine Configuration window, select Typical (recommended).

4) In the Install operating system from: window, choose Use ISO image and provide the path to the ISO image you downloaded. Fill in the required information in the Personalize Linux section.

Warning: This password will be used to log into the VM, so make sure to remember it!

5) In the Virtual Machine Name window, leave everything as is, and click Next.

6) In the Disk Size section, choose a size of at least 32 GB and select Store virtual disk as a single file.

7) In the final section, click Customize Hardware and configure the following settings

8) Click close and after the configuration click Finish to create your new VM.

Some notes for the Ubuntu installation:

Finally click Install.

After a while, hopefully your VM is ready! Make sure you can boot and log into it before proceeding to the next section.

Enable Clipboard Sharing (Host ↔ Guest)

To enable clipboard sharing between the host and the guest virtual machine, open a terminal in the VM and run:

sudo apt update && sudo apt install open-vm-tools open-vm-tools-desktop
sudo systemctl enable --now open-vm-tools

Afterwards, log out of your session and log back in. Clipboard sharing between the host and the guest should now be enabled.

Kernel Compilation & Installation

Warning: All the steps of this sub-section must be run inside the VM!

Preparation

Boot up the VM and from a shell inside the VM execute the following commands to install the required dependencies for building the kernel:

sudo apt update
sudo apt install build-essential git bc python3 bison flex rsync libelf-dev libncurses-dev dwarves libdw-dev gawk libssl-dev openssh-server htop universal-ctags python3-ply python3-git

Next, clone the source code for Linux kernel v7.2

git clone --depth=1 --branch v7.2 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-7.2
cd linux-7.2

Warning: all subsequent commands from this guide should be run from the root directory of the kernel source!

Read the first lines of the Makefile to verify you downloaded the correct kernel version:

head -n5 Makefile
# SPDX-License-Identifier: GPL-2.0
VERSION = 7
PATCHLEVEL = 2
SUBLEVEL = 0
EXTRAVERSION =

Kernel Configuration

Before building the kernel, you must configure it by selecting the features, drivers and other options that will be included in the compiled kernel. This configuration is stored in the .config file in the root directory of the kernel source.

We will remove any existing configuration files and create our own based on the configuration file of the kernel currently in use.

Warning: it would be a good idea to run this step using a stable stock kernel, such as the one provided by default by your distribution, to ensure you proceed with a proper configuration.

make mrproper
make olddefconfig

A .config file should now be visible in the current directory. This file holds kernel build options, each set to a specific value. You can view its contents with cat .config. There are two main approaches to edit the configuration file:

scripts/config --set-str <option> <value>  # set an option
scripts/config --state <option>            # read an option

Make the following changes to your configuration file:

scripts/config --set-str CONFIG_LOCALVERSION "-k22"
scripts/config --set-str SYSTEM_TRUSTED_KEYS ""
scripts/config --set-str SYSTEM_REVOCATION_KEYS ""

The first command gives your kernel a custom name to distinguish it from other installed kernels. You should expect your custom kernel to have a name similar to 7.2-k22. The lattercommands instruct the kernel not to enforce built-in signature verification of modules/firmware against specific trusted keys.

Finally, run the following command to disable unused kernel modules from your configuration file, reducing kernel compilation time:

yes '' | make localmodconfig

Kernel Build

Warning: run this step as non-root user!

Build the kernel using all cores allocated to the VM with:

make -j`nproc`

Depending on the amount of available cores and the capabilities of your machine, this step may take a while.

Kernel Installation

Install your custom kernel and kernel modules with:

sudo make modules_install && sudo make install

If all goes well, you must be able to verify that the following 3 files are in /boot:

ls /boot | grep "k22"
initrd.img-7.2-k22
System.map-7.2-k22
vmlinuz-7.2-k22

Boot the Custom Kernel

Before rebooting the machine, we need to modify the bootloader’s (GRUB) configuration file to allow us to select the kernel we want to boot with.

Open /etc/default/grub as root with a text editor and:

Save and close the file and then run:

sudo update-grub

Reboot the machine and from the GRUB menu select Advanced Options for Ubuntu . Choose the kernel post-fixed with -k22. You can always select the Ubuntu option from the GRUB menu to boot with the stock kernel, which is recommended for development.

Once the VM boots up, verify you are running your custom kernel with:

uname -r
7.2-k22

Option-2: Set up a VM with KVM/QEMU (Ubuntu users)

This setup option is an alternative for those who wish to obtain access to tools like a debugger, and it can only be used if your host machine is GNU/Linux.

Enable Hardware Virtualization

We need to enable hardware level virtualization support via the motherboard UEFI/BIOS firmware settings.

Visit your motherboard manufacturer’s website (or your laptop’s manufacturer’s website if you are using a laptop) and find the page for your specific device. In the manuals of the manufacturer you will find the way to enter the UEFI/BIOS menu settings of your device and edit them. Look for settings with names like Virtualization Support, VMX, Enable virtualization. It should be a boolean toggle (true/false or enabled/disabled) which you want to ensure is turned on (enabled, true).

You can find out if you correctly enabled virtualization by running this command on your host OS and making sure the output is non-zero:

grep -E -c '(vmx|svm)' /proc/cpuinfo

Enable the KVM Module on your Host Kernel

To check if the KVM kernel module on the host OS is properly installed, run the following command:

lsmod | grep kvm

If the output looks like this, (perhaps with kvm_amd instead of kvm_intel) you can skip the rest of this step.

kvm_intel             286720  0
kvm                   667648  1 kvm_intel

After you are done with this, rerun the command lsmod | grep kvm and ensure the output shows the kvm module and one of { kvm_intel, kvm_amd }. If it does, move to the next step. If not, please contact us so we can help you.

Install QEMU

Unlike the normal setup, where we use VMware, we need to install the virtual machine environment we will be using.

We will be installing QEMU, which has builtin support for KVM and debugging. QEMU is commonly used among kernel developers as it can be quickly launched and has near native performance allowing for quick iteration.

We highly encourage exploring these tools outside of what we explain in here, as they are used very often in the industry and having extensive knowledge of them can significantly enhance the OS level development experience.

If your host is on Ubuntu, you can install the above software by running:

sudo apt install qemu-system-x86 cpu-checker

Run the following command and ensure that the output says everything is fine. Otherwise, you either made a mistake in the previous steps. Retry them. If even after that your setup does not work, contact us on Piazza so we may help you.

kvm-ok

Kernel Configuration & Installation

A big benefit of this tutorial is that kernel compilation will be done in the host computer instead of inside the virtual machine which will allow us to save some time during compilations.

Download an Ubuntu ISO Image

We will work with Ubuntu 26.04.1 LTS

Create a Virtual Disk

We will create our virtual disk, which, we will be using throughout the rest of the tutorial.

In a directory that you can easily recall (do NOT use the linux kernel directory as you may want to clean it later), run the following command

qemu-img create -f qcow2 rootdisk.qcow2 32G # replace 32G with however much storage you want

We recommend using at least 20G, and 32G is probably as much as you will need. (We have run into problems with running out of disk space, and thought something is wrong with our kernel build. Thus, always keep in mind the amount of storage you have allocated.)

Boot the Custom Kernel on the VM

We will use the virtual disk created during the previous step to run our virtual machine in TTY mode. That means we will only get console output and have no access to UI.

This is a good format for quick development iteration and testing.

Follow the instructions from Option-1 above to compile a fresh copy of the kernel on your host machine. Then, in the root directory of the kernel, run the following command, substituting any placeholders for what you want to use:

qemu-system-x86_64 -m 4096  \
                     -drive format=qcow2,file=./rootdisk.qcow2 \
                     -enable-kvm \
                     -smp 8 \
                     -audio driver=pa,model=virtio \
                     -cdrom <path-to-your-ISO> # substitute the path to your ubuntu ISO image

This should now start outputting kernel boot log and finally, bring you to the Ubuntu installation wizard.

Install a Fresh OS into the VM

In this step you will need to follow the steps the Ubuntu installer has in order to install Ubuntu in the virtual machine. For more information, see this.

At this point, you should now have a fully functioning virtual machine running Ubuntu 26.04.1 LTS.

Boot a Custom Kernel

To boot a custom kernel build in our VM, what we need to do is go to the root path of the linux kernel clone and from there run the following command:

qemu-system-x86_64 -kernel ./arch/x86/boot/bzImage \
                     -append "nokaslr console=ttyS0 root=/dev/sda2" \
                     -nographic  \
                     -m 4096  \
                     -drive format=qcow2,file=<path-to-your-virtual-disk>  \
                     -enable-kvm  \
                     -smp 8

You should now see TTY output in the terminal, showing kernel logs and then be prompted for a login user and password. Congratulations, you have now run your custom kernel build in a virtual machine!

Debugging

With this setup, we also have the ability to connect to our virtual machine with a GDB session and debug the kernel. To do that, make sure you have gdb installed.

We also want to make sure our kernel is configured with debugging features enabled. There a bunch of them you can take a look at. For now we will need the following:

./scripts/config --enable CONFIG_GDB_SCRIPTS # enabled GDB scripts
./scripts/config --enable CONFIG_DEBUG_INFO # compile with debug info
./scripts/config --enable CONFIG_DEBUG_INFO # compile with debug info
./scripts/config --enable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT # compile with debug info
./scripts/config --disable CONFIG_DEBUG_INFO_REDUCED # compile with full debug info
./scripts/config --enable CONFIG_FRAME_POINTER # keep frame pointer info
./scripts/config --disable CONFIG_RANDOMIZE_BASE # allow for disabling kernel address space layout randomization (kaslr)

We then need to create the kernel GDB scripts. To do that, run the following command:

make scripts_gdb
echo "add-auto-load-safe-path /path/to/linux-build" > ~/.gdbinit # replace the placeholder path with the path to your linux build

Then just start you virtual machine, making sure to add the -s and -S flags.

qemu-system-x86_64 -kernel ./arch/x86/boot/bzImage \
                     -append "nokaslr console=ttyS0 root=/dev/sda2" \
                     -nographic  \
                     -m 4096  \
                     -drive format=qcow2,file=<path-to-your-virtual-disk>  \
                     -enable-kvm  \
                     -smp 8       \
                     -s           \
                     -S           # stop and wait for GDB connection

Then, start a GDB instance in a separate terminal. To do that, go to your kernel root folder and run

gdb ./vmlinux

A gdb terminal will now open. In that terminal, type

(gdb) target remote :1234

This should now connect you to kernel instance running in the virtual machine. You can make sure that everything is correct, by running

(gdb) continue

The virtual machine should now start booting. For more information regarding how to use GDB to debug a program, look here.

GDB is going to be an exceptionally useful tool throughout your kernel development journey. It can save a lot of time and help understand a lot of conditions. There are numerous other tools one can use to make kernel development and debugging easier but we won’t go into too much detail regarding those. Feel free to explore and use anythingyou deem useful.

Mounting your Virtual Disk

In some cases, like when installing modules or headers from a custom kernel build, it’s necessary for the host to be able to access the filesystem in the virtual disk. To do that, we will use an NBD (network block device).

First we need to enable the NBD module on the host. To do that, run:

sudo modprobe nbd

We then need to connect our virtual disk to an NBD device. We will use the qemu-nbd tool to do that. Run:

sudo qemu-nbd -c /dev/nbd0 <path-to-your-virtual-disk>

You can now mount the partitions of your virtual disk to your host:

sudo mount /dev/nbd0p2 /mnt # mount the second partition of the virtual disk

To install headers to your VM, you need to run the headers_install rule AND modify the path of the installation as shown below:

make headers_install INSTALL_HDR_PATH=/mnt/usr

You can install modules to your VM in a similar way, by running:

make modules_install INSTALL_MOD_PATH=/mnt

You now need to unmount your nbd device and then disconnect it from the virtual disk (you VM cannot boot with its disk mounted on the host):

sudo umount /dev/nbd0p2
sudo qemu-nbd -d /dev/nbd0

You can now safely boot back into your VM and the new headers as well as the new modules should be properly installed.

Option-3: Set up a VM with UTM (MacOS users)

Finally, this option covers the setup for MacOS users, including a debugger setup since it will come in handy for most of the assignments.

Install UTM Virtual Machine software

You can install UTM with brew (recommended) or from its website.

For installation using brew, just type in you terminal.

brew install utm

Do not obtain the framework from the App Store, as it is not free there for technical reasons. The versions that can be installed with the recommended methods are more than complete and fit for the purposes of this course.

Download an Ubuntu ISO Image

We will work with Ubuntu 26.04.1 LTS

VM Setup

Once downloaded, open UTM. On the top, you will see a “+” button, press it and choose Virtualize. Emulation is not forbidden but neither necessary. If you choose to emulate, make sure you have the ISO for the architecture of your choosing.

When asking for an Operating System, choose “Linux”. You are recommended to provide at least:

For Display Output leave as default.

On the next page, choose:

Provide at least 32 GB of disk space, but more is recommended. You also do not need to choose to share a directory, unless you want to.

Lastly, name your VM, save it and you are good to go!

Booting and Configuration

Once you boot your VM you will have to install some packages, the kernel directory and configure it. Follow the instructions from this part of the tutorial.

GDB setup

Sometimes, when things don’t go as planned (which will happen very often), it might will be hard to troubleshoot and debug your kernel. This is why we recommend setting up your VM to work with GDB sessions. If you do not have gdb, you can install it via brew.

In UTM, select your vm from the side bar. On the top right choose to “Edit selected VM”. Under the QEMU option go to Arguments and scroll to the bottom. We will add two argumentsthat will tell QEMU we want to

Choose New and add the flags above as two different commands. Note that the flag should be on the “dark bar” since it is not completely specified by utm. Click save and you aredone.

For the rest, follow this part of the tutorial

Our First Kernel Patch

We will now make our first Linux kernel patch to get used to the kernel development workflow. We need the kernel to print a Hello World message whenever the VM boots (i.e., when the kernel starts). This message should be displayed in the kernel message buffer, used by the kernel to display useful messages, such as:

To do this, we will edit the start_kernel() function in the init/main.c file. This is the C entry point of the Linux kernel and is called by the architecture-specific assembly start up code, after the CPU and basic hardware have been initialized. The function “ends” with a call to rest_init(), which performs the remaining initializations and calls into the scheduler. Therefore rest_init() should never return.

We will print a log message in start_kernel(), just before rest_init() is called. The logging function we will use is called printk(). It works similar to printf but its messages are all printed to the kernel message buffer.

Insert the following line of code just before the call to rest_init():

printk(KERN_INFO "I miss Alexis Delis\n");

Warning: this step should be performed in your custom kernel tree inside the VM!

Make sure you save the file, re-build and re-install the kernel with the following commands:

make -j`nproc`
sudo make modules_install && sudo make install

Note: You do not need to re-configure the kernel every time you make a change. However you should re-compile any changed files and re-install the modified kernel using the commandsabove whenever you are ready to test your changes.

Reboot the machine and once again select your custom -k22 kernel from the GRUB menu. The new kernel may show a -dirty suffix, which is normal. Run the following command to searchthe kernel logs for your custom message:

sudo dmesg | grep "Delis"
[    0.148119] I miss Alexis Delis

Authors

K22 Fall ‘26 crew!