Debugging an OpenWrt/LEDE kernel using KGDB over serial

When trying to get sound output up and running on my LinkIt Smart MT7688 I needed to debug the Linux Kernel. I am using the master branch of the OpenWrt/LEDE distribution and therefore wanted to configure the sound card using the simple-card device tree binding .

Since I needed to debug loadable Kernel modules (LKM), the process is a little more involved than it could be. There are three aspects to take care of:

1. The Kernel needs to be compiled with the Kernel debugging backend KGDB and debugging symbols activated.
2. A suitable GDB front end has to be built. This needs to have Python support enabled in order to make use of the Kernel GDB scripts.
3. The LKM that need to be debugged have to be built with optimization disabled and debugging symbols enabled.

At least one serial (UART) interface is needed on the device under test that will be exclusively used by the debugger. Of course Python needs to be installed on the host machine.

Configuring the Kernel

Enter the Kernel configuration command line tool by running make kernel_menuconfig. Search for and set the following options:

• CONFIG_DEBUG_INFO=y
• CONFIG_DEBUG_INFO_DWARF4=y
• # CONFIG_DEBUG_INFO_REDUCED is not set
• CONFIG_GDB_SCRIPTS=y
• CONFIG_KGDB=y
• CONFIG_KGDB_SERIAL_CONSOLE=y
• Additionally the Kernel needs to be told what serial interface to use for the debugger. In my case I added kgdboc=ttyS1,115200 to CONFIG_CMDLINE.

Configuring the GDB front end

In the distribution configuration command line tool (make menuconfig) make sure that the GDB is being built as part of the toolchain. Then in the Makefile for the GDB which can be found at toolchain/gdb/Makefile the following changes need to be made:

• Replace --without-python with --with-python in the HOST_CONFIGURE_ARGS.
• In the procedure Host/Install append $(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory install and in the procedure Host/Clean prepend $(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory uninstall.

Now everything needs to be built for the first time using make.

Adjust the compile flags for UUT

Say the unit under test is the I²C driver for Ralink/Mediatek MIPS SoCs which can be found at build_dir/target-mipsel_24kc_musl-1.1.15/linux-ramips_mt7688/linux-4.4.24/drivers/i2c/busses/i2c-mt7621.c. Now the Makefile that resides in the same directory as the source file needs to be modified. The line CFLAGS_i2c-mt7621.o := -O0 -ggdb3 is added which disables optimization and enables the most verbose debugging information.

After the modification everything can be compiled again and the device under test can be flashed with the newly generated image.

Debugging procedure

For convenience it makes sense to add the directory that the GDB front end binary resides in to the PATH variable. In my case this directory is staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin. In a terminal with that variable set we navigate to the Kernel base directory in build_dir/target-mipsel_24kc_musl-1.1.15/linux-ramips_mt7688/linux-4.4.24. There we start the GDB front end with vmlinux as the argument. It might be necessary to explicitely allow the GDB front end to load the Python scripts using the command add-auto-load-safe-path.

On the device under test the Kernel execution needs to be manually interrupted by issuing the command echo g > /proc/sysrq-trigger. An intervention from the GDB front end is not possible when debugging the Kernel. Now the Kernel is listening on the configured (via Kernel command line) serial interface for the GDB front end.

In the GDB front end session the baudrate needs to be configured using set serial baud 115200. After that we connect to the target using the command target remote /dev/ttyXXX where XXX is the serial interface identifier on the host machine. The serial interface being the one that is connected to the debugging serial interface on the target. Now the GDB front end should be attached to the Kernel debugger and e.g. breakpoints can be placed and excecution continued.

Debugging an LKM which is not loaded yet

The only way to debug the initialization of a module I am aware of is to export the initialization function symbol. This can be done by using the macro EXPORT_SYMBOL_GPL or EXPORT_SYMBOL. Of course this requires recompiling of the kernel.

When the GDB front end is attached and the target’s Kernel execution is stopped, the command lx-symbols is issued which detects all currently loaded modules and loads their symbols. Furthermore we need to assure that pending breakpoints are allowed by executing set breakpoint pending on. Now the breakpoint itself can be set in the usual way and the Kernel execution can be resumed. When the module is inserted now the execution will stop at the beginning of the init function. The next steps are discussed in the following section.

Debugging an LKM which is already loaded

As I couldn’t find a way to tell the Kernel build system to not strip the LKM binaries (*.ko), in the next step the object file with all the debug symbols needs to be loaded with the correct address offset which is determined at runtime.

If a new kernel module has been inserted the command lx-symbols needs to be run again. The command lx-lsmod lists all the currently loaded Kernel modules with their base addresses. After finding the correct address for our LKM we add the object file as a symbol file using add-symbol-file path_to_the_object_file address. In my case it boils down to add-symbol-file drivers/i2c/busses/i2c-mt7621.o 0x12345678. Now all symbols from that object file are known and can be used for setting breakpoints, etc.

One last thing: I found the GDB dashboard to be really handy!