- Decent ARM processor at 1ghz
- 256mb of ram
- 512mb of flash
- Plenty of I/O
- A nice power controller module
- And a very nice dual channel 802.11 solution
- A tidy buildroot platform more or less ready to extend
Anyway, I am building some IoT applications (with AWS Greengrass ) to explore continuous security patterns, over the air upgrade and mesh architectures.
As is the case with lower level hardware solutions, I'm extending a linux drivers and introducing problems along the way. Eventually needing to do low level startup debugging. Not much documentation on this so I figured I'd dig in and work it out.
- Build the platform with debugging enabled
- Load this image onto the target as usual
- Modify boot args to start up and enter kdb
- Then through the console, flip over to kgdb
- And launch gdb-multiarch inside the buildroot VM
Steps to get kernel debugging enabled
This all assumes you are familiar with ChipPro buildroot builds as shown here:
Configure the build to include kernel debugging
Add these entries to the linux-new.config (inside your clone of gadget-buildroot)
CONFIG_DEBUG_INFO=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_DEBUG_RODATA is not set
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
Rebuild the kernel from scratch
scripts/build-gadget make clean
scripts/build-gadget make xxx_defconfig
scripts/build-garget make -s
Load the newly built image onto system flash
(following instructions from gadget-buildroot)
Modify boot args to enter kdb on early boot
Now, assuming you have a serial port attached to your build system, you should be able to connect to the serial console to watch boot, etc. For me on a Mac this is:
screen /dev/tty.usbserial 115200
Here is where you can interrupt boot sequence at the firmware load, modify boot environments, etc. You can either interrupt the boot sequence (e.g. if your code updates panic the kernel early). or you can use fw_printenv, fw_setenv which are commands in the target's image.
The environment parameters (fw_printenv) should show one entry like this:
bootargs=root=ubi0:rootfs rootfstype=ubifs rw ubi.mtd=4 lpj=5009408 ubi.fm_autoconvert=1 quiet
Now, modify this entry (using fw_setenv) to enter kdb, using the serial console on boot:
root=ubi0:rootfs rootfstype=ubifs rw ubi.mtd=4 lpj=5009408 ubi.fm_autoconvert=1 kgdboc=ttyS0,115200 kgdbwaitAt this point if all is well, when you reset the device it should enter kdb.
Use gdb to debug the kernel
Several things need to happen here. You need to get the serial console access inside the docker container that does the build. Depending on the VM you are using, the steps to get the USB serial port visible in the buildroot guest vary. Since I'm using VirtualBox on a Mac, I wound up enabling USB on the guest and restarting it. Native docker support will be a bit simpler.
Anyway, once the port is available to the guest, you need to alter scripts/build-gadget to include the device. Something like this on the docker run command in the build-gadget script (the --device line):
Now, you will need to enter the guest interactively. The best way to do this is (from the gadget-buildroot top level directory)docker run \--rm \--env BR2_DL_DIR=/opt/dlcache/ \--volume=gadget-build-dlcache:/opt/dlcache/ \--volume=gadget-build-output:/opt/output \--volume=${IMAGE_DIR}:/opt/output/images \--volume=${GADGET_DIR}:/opt/gadget-os-proto \--name=gadget-build-task \-w="/opt/gadget-os-proto" \--device=/dev/ttyUSB0 \
TERM=yes ./scripts/build-gadget bashNow you should install screen and gdb-multiarch (TODO this should be part of the build-container)
apt update
apt-get install screen gdb-multiarchNow, connect to the serial console from inside the guest
screen /dev/ttyUSB0 115200And boot up the device to kdb. And then flip the console over to kgdb (by entering 'kgdb' at the kdb prompt). At this point you need terminate screen (terminate, not detach -- otherwise complaints about port in use).
Now debug away:
In the guest, change over to the linux directory of the build. This is something like /opt/output/build/linux-... And then start up gdb in cross platform mode:gdb-multiarch
set arch arm
file vmlinux
target remote /dev/ttyUSB0That should do it, you are in an early boot intercept using a remote debugger from inside a guest OS that has all the necessary source!
Happy debugging!
