How can you verify that your Linux kernel image will boot on real hardware? Booting your image with virtualization technologies can’t provide you with an absolute guarantee that your software will run on metal. However, QEMU and Libvirt give you enough capabilities to conduct a quick sanity check and get rid of most bugs at the development stage.
In this article, we explain how you can debug your Linux kernel and its modules during runtime. This article will be useful for Linux kernel developers who want to speed up the time to market of their software.
We’ll provide you with step-by-step instructions on how you can find errors in your Linux kernel module or kernel image at the development stage. Our method includes three levels:
- Debugging without source code
- Debugging with source code
- Debugging with third-party modules
You can either use one of these levels or dig deep and go through the whole process of kernel debugging.
To conduct kernel debugging, you should prepare the necessary environment. First of all, you need to set up the guest target.
The guest is a Libvirt or QEMU-style virtual machine that we’ll debug. The host is an operating system where debugging is taking place.
To create an environment, we use Ubuntu 16.04 as the host operating system and Ubuntu 16.04 as the guest operating system.
However, you can also use other guest Linux distributions. Just make sure to install the corresponding packages beforehand.
Here’s what you need to install for debugging:
- QEMU as a hypervisor and emulator for hardware virtualization
- Libvirt as a daemon and a toolkit for managing platform virtualization
- GDB as a project debugger
- A QEMU virtual machine (in our case, it’s Ubuntu 16.04 under the name 04)
Debugging without the source code is the simplest way to quickly debug the kernel if you need to look at the call stack and variable values. You can verify the kernel image in this way:
- Configure Libvirt on the host operating system with the following command:
In your editor of choice, you can open the file with virtual machine settings. Find the
domain tag and add the following option:
Save the file and exit the editor. Now you need to restart the virtual machine (if it’s already running).
The -s parameter means that QEMU will open port 1234 for debugging by default. If you need to open other ports, you can set them in the domain XML file:
The specified port should open after the virtual machine starts.
- Disable Kernel Address Space Layout Randomization (KASLR) on the guest. To do this, you need to apply changes directly to the GRand Unified Bootloader (GRUB) configuration. Open /etc/default/grub and add nokaslr to the GRUB_CMDLINE_LINUX_DEFAULT variable:
Update GRUB with this command:
and reboot the guest operating system.
- The Linux kernel packages do not include debug symbols, as they’re stripped from binary files at build time. However, you need to install debug symbols to let GDB assist you during debugging. To install debug symbols for the existing kernel on the guest, see the detailed description here.
Here how we’ll install debug symbols on Ubuntu:
You can manually download and install the ddeb package from here as an example. However, be careful to install a package with the compatible kernel and architecture.
Here’s how to install debug symbols on Fedora Linux:
You can find a complete description of how to install debug symbols on Fedora Linux here.
- Copy the /usr/lib/debug/boot/vmlinux-$(uname -r) file from the guest to any folder on the host so GDB can use it. In our case, we have the following file:
- Run the QEMU virtual machine on the host and connect it to the debugger:
Here’s an example of Linux kernel debugging for Ubuntu 16.04 (4.13.0-36):
Source code allows the debugger to associate events and circumstances in program execution with their corresponding location. So using the source code, you can run the (gdb) list command without errors like “No such file or directory.” If you have a source code file, make it available to GDB for kernel debugging in the following way:
- Load the source code to the host (see detailed description here). You can read how to download the source code from the git repository here.
- Connect the source code to GDB with the
gdb ./vmlinux-4.13.0-36-generic (gdb) set substitute-path /build/linux-hwe-4GXcua/linux-hwe-4.13.0 /media/veracrypt2/linux_kernel_new/ubuntu-xenial
If you look at the previous logs, you can see the path to the source code in the debug symbols that is
Substitute the original source code path with the path to our source code using the
set substitute-path command. In our case, the new path is
This method is for those who want to debug a module that’s not included in the official Linux kernel version. For instance, we’ll try to debug our own module.
- Build the module with debug symbols on the guest and install it. To build the module with debug symbols, add the -g flag (EXTRA_CFLAGS += -g) to the compiler options. Copy the built module to the host folder together with the debug symbols (or to the folder with the source code). In our case, we get this:
- Build the GDB scripts on the host so we can debug modules that are loaded. Unfortunately, scripts are provided only with the whole kernel, so we need to stop the building process after we get the debian/build/build-generic/scripts/gdb/linux/constants.py file (you can find more information about kernel building here).
If you want to continue building after interruption, make an additional link to the script:
- Permit the GDB to load. Create the ~/.gdbinit file as a user on the host:
- The file with debug symbols should be named vmlinux to make our scripts runnable, so rename this file on the host:
- Run GDB and make sure that everything is okay with the scripts. We can also check if it’s possible to set breakpoints in our module and debug it (see comments in the script). The breakpoint makes the debugger stop execution after a specified function or line number in the source file.
The line “no module object found for” means that there are no other modules that can be loaded. If the debugger can’t find source code files, you can manually add links to the source code for the modules. If any modules are available on the machine, you’ll see the following:
- If you want to check other modules, you can add links to the source code for them in the same way as we did in the first step:
Debugging a Linux kernel image or module is a complex task that many developers face. Using such virtualization technologies as QEMU and Libvirt allow them to detect and fix the majority of defects at the development stage.
In this article, we showed you three levels of debugging: debugging with and without source code and debugging with the help of third-party modules. Depending on the task at hand, you can either debug your kernel image or module on one of these levels or go through the whole process.
Here at Apriorit, we have a team of passionate kernel and driver developers who will be glad to assist you in creating your own software solution for the Linux platform.