When trying to get sound output up and running on my LinkIt Smart MT7688 I needed to debug the Linux Kernel. I am using the master branch of the OpenWrt/LEDE distribution and therefore wanted to configure the sound card using the simple-card device tree binding.
Since I needed to debug loadable Kernel modules (LKM), the process is a little more involved than it could be. There are three aspects to take care of:
- The Kernel needs to be compiled with the Kernel debugging backend KGDB and debugging symbols activated.
- A suitable GDB front end has to be built. This needs to have Python support enabled in order to make use of the Kernel GDB scripts.
- The LKM that need to be debugged have to be built with optimization disabled and debugging symbols enabled.
At least one serial (UART) interface is needed on the device under test that will be exclusively used by the debugger. Of course Python needs to be installed on the host machine.
Configuring the Kernel
Enter the Kernel configuration command line tool by running
make kernel_menuconfig. Search for and set the following options:
# CONFIG_DEBUG_INFO_REDUCED is not set
- Additionally the Kernel needs to be told what serial interface to use for the debugger. In my case I added
Configuring the GDB front end
In the distribution configuration command line tool (
make menuconfig) make sure that the GDB is being built as part of the toolchain. Then in the Makefile for the GDB which can be found at
toolchain/gdb/Makefile the following changes need to be made:
- In the procedure
$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory installand in the procedure
$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory uninstall.
Now everything needs to be built for the first time using
Adjust the compile flags for UUT
Say the unit under test is the I²C driver for Ralink/Mediatek MIPS SoCs which can be found at
build_dir/target-mipsel_24kc_musl-1.1.15/linux-ramips_mt7688/linux-4.4.24/drivers/i2c/busses/i2c-mt7621.c. Now the Makefile that resides in the same directory as the source file needs to be modified. The line
CFLAGS_i2c-mt7621.o := -O0 -ggdb3 is added which disables optimization and enables the most verbose debugging information.
After the modification everything can be compiled again and the device under test can be flashed with the newly generated image.
For convenience it makes sense to add the directory that the GDB front end binary resides in to the
PATH variable. In my case this directory is
staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin. In a terminal with that variable set we navigate to the Kernel base directory in
build_dir/target-mipsel_24kc_musl-1.1.15/linux-ramips_mt7688/linux-4.4.24. There we start the GDB front end with
vmlinux as the argument. It might be necessary to explicitely allow the GDB front end to load the Python scripts using the command
On the device under test the Kernel execution needs to be manually interrupted by issuing the command
echo g > /proc/sysrq-trigger. An intervention from the GDB front end is not possible when debugging the Kernel. Now the Kernel is listening on the configured (via Kernel command line) serial interface for the GDB front end.
In the GDB front end session the baudrate needs to be configured using
set serial baud 115200. After that we connect to the target using the command
target remote /dev/ttyXXX where
XXX is the serial interface identifier on the host machine. The serial interface being the one that is connected to the debugging serial interface on the target. Now the GDB front end should be attached to the Kernel debugger and e.g. breakpoints can be placed and excecution continued.
Debugging an LKM which is not loaded yet
The only way to debug the initialization of a module I am aware of is to export the initialization function symbol. This can be done by using the macro
EXPORT_SYMBOL. Of course this requires recompiling of the kernel.
When the GDB front end is attached and the target’s Kernel execution is stopped, the command
lx-symbols is issued which detects all currently loaded modules and loads their symbols. Furthermore we need to assure that pending breakpoints are allowed by executing
set breakpoint pending on. Now the breakpoint itself can be set in the usual way and the Kernel execution can be resumed. When the module is inserted now the execution will stop at the beginning of the init function. The next steps are discussed in the following section.
Debugging an LKM which is already loaded
As I couldn’t find a way to tell the Kernel build system to not strip the LKM binaries (
*.ko), in the next step the object file with all the debug symbols needs to be loaded with the correct address offset which is determined at runtime.
If a new kernel module has been inserted the command
lx-symbols needs to be run again. The command
lx-lsmod lists all the currently loaded Kernel modules with their base addresses. After finding the correct address for our LKM we add the object file as a symbol file using
add-symbol-file path_to_the_object_file address. In my case it boils down to
add-symbol-file drivers/i2c/busses/i2c-mt7621.o 0x12345678. Now all symbols from that object file are known and can be used for setting breakpoints, etc.
One last thing: I found the GDB dashboard to be really handy!