Arroyo.cc / plan9 / solo-factotum / A factotum in your USB port, part 2

In the last post, I established a process for building a free-standing binary with Plan 9's C compiler toolchain, flashing that binary onto an STM32 microcontroller using st-link, and inspecting the state of the microcontroller using gdb. It is a primitive, but sufficient starting condition for the iterative process of building. We got as far as blinking an LED on the NUCLEO L432KC development board.

Debugging with acid(1)

In the previous post, I used the GNU debugger (gdb) to debug the device. This was serviceable, but since gdb does not understand Plan 9's a.out(6) format for executables, I was limited to inspecting disassembly and manually poking at registers and stack space, like a caveman. While that works for small programs like the blinking LED demo, it will get tiresome as I move on to more complex software.

So I wrote gdbfs(4), a file server which connects to the gdb stub provided by the stlink project, and binds a file system over /proc which makes the target appear like any other process. You can use acid(1) and db(1) for debugging, or snap(1) to save the state of the process to a file (which can then be debugged later with snapfs(4), or shared with others).

You can export the files to the network, so someone on the other side of the room (or the world) can debug your microcontroller.

While I wrote gdbfs for the sake of this project, implementations (colloquially called gdb stubs) of the GDB Remote Serial protocol are quite pervasive, so this server is useful beyond microcontrollers. The BlackMagic probe is an implementation of this idea in hardware, and supports many debug targets through SWD or JTAG. Since many VM hypervisors also have a gdb stub, you can use this, for example, to debug a Plan 9 kernel running in a QEMU hypervisor with the -s option:

# we're running from drawterm, make the (linux) terminal's localhost reachable
% bind /mnt/term/net /net
% gdbfs -t /amd64/9pc64 localhost
# /proc/1 has become the process dir for the target
% acid -l kernel 1
acid: procs()
0xffffffff80056ca8 1: bootrc (null) pc 0xffffffff8023467c Await (Wakeme) ut 0 st 0 qpc 0x00000000
0xffffffff8005d7e8 3: pager (null) pc 0x00000000 Idle (Wakeme) ut 0 st 1 qpc 0x00000000
0xffffffff808acc38 4: mouse (null) pc 0x00000000 kproc (Wakeme) ut 0 st 1 qpc 0x00000000
0xffffffff808b15b8 5: xhcirecover (null) pc 0xffffffff80223f9c kproc (Wakeme) ut 0 st 2 qpc 0x00000000
0xffffffff808b3e98 295: closeproc (null) pc 0xffffffff80223f9c kproc (Wakeme) ut 0 st 15362 qpc 0xffffffff8021b43e
... snip ...
acid: stacks()
=========================================================
0xffffffff8090f218 36: #l0rx (null) pc 0x00000000 kproc (Wakeme) ut 0 st 4 qpc 0x00000000
gotolabel()+0x0 /sys/src/9/pc64/l.s:576
procswitch()+0x50 /sys/src/9/port/proc.c:158
sleep(r=0xffffffff800491e0,arg=0xffffffff800491e0,f=0xffffffff801ac96f)+0x185 /sys/src/9/port/proc.c:857
rxproc()+0x1fb /sys/src/9/pc/ethervirtio.c:321
linkproc()+0x19 /sys/src/9/port/proc.c:1579
0xffffffff8090f0f8 ?file?:0
... snip ...

This would be useful for debugging kernel issues that would disrupt other means of access (network, graphics, etc). This is similar to rdbfs(4), which allows a remote 9front kernel (with the appropriate boot option) to be debugged over a serial line. The gdbfs program is now part of 9front.

disassembly of Thumb code in acid(1)

Although 9front includes a Thumb compiler and linker, there is no 9front kernel that uses it; there are some 32-bit ARM kernels, but platforms which can only execute Thumb code tend to not have MMUs, and it would be very challenging to port 9front to a system without one. As such, the Thumb toolchain, brought over from Inferno many years ago for the purpose of building homebrew software for the Gameboy, is under-used, so there may be gaps here and there.

With gdbfs(4) I am able to puppeteer a gdbserver with the plan 9 debugging tools:

% bind /mnt/term/net /net
% gdbfs -t app/blink/t.out tcp!localhost!4242
% acid 1
/proc/1/text:ARM plan 9 boot image
/sys/lib/acid/port
/sys/lib/acid/arm
acid: *PC
0x0800043c
acid: asm(_main)
_main 0x08000400	AND.NE	R0,R0,R4
_main+0x4 0x08000404	MOVM.EQ	[R0,R10],(R0)

But something is wrong. The disassembly of _main should look the same as the pseudo-assembly output by the linker's -a flag:

% tl -T0x08000000 -R0 -D0x10000000 -a -H0 \
	-o app/blink/stm32l4.bin stm32l4/lib/libc.a app/blink/blink.t
... snip ...
 08000400:		(270)	TEXT	_main+0(SB),R4,$-4
 08000400: 0000491e	(272)	MOVW	$setR12+0(SB),R1
 08000402: 0000468c	(273)	MOVW	R1,R12
 08000404: 0000f000 0000f81e	(273)	BL	,_init+8000444(BRANCH)

Let's compare the bytes:

acid: dump(_main, 3, "x")
0x08000400: 0x491e 
0x08000402: 0x468c 
0x08000404: 0xf000 
0x08000406: 0xf81e

the bytes match, but acid is interpreting them as two ARM instructions instead of 4 Thumb instructions. When a binary is produced by tc and tl, it is supposed to include a table of locations in the text segment which are thumb-encoded. This table is built in /sys/src/cmd/tl/asm.c:/^asmthumbmap/ . Each entry has the format

struct pcentry{
	long start;
	long stop;
};

I ended up writing a very minimal program that tries to disassemble an arbitrary address:

fd = open(path, OREAD);
if(fd < 0)
	sysfatal("open: %r");
if(crackhdr(fd, &hdr) == 0)
	sysfatal("crackhdr: %r");
if(hdr.type != FARM && hdr.type != FARMB)
	sysfatal("not an arm binary");
machbytype(hdr.type);
if((symmap = loadmap(0, fd, &hdr)) == nil)
	sysfatal("loadmap: %r");
if((*machdata->das)(symmap, addr, 'i', buf, sizeof buf) < 0)
	sysfatal("das: %r");
print("%s\n", buf);

And running that under a debugger. Eventually I looked in the thumbpctab function of libmach, which reads the table of Thumb ranges:

acid: *(*thumbpctab:tab\X) # tab.start
0x08000800
acid: *(*thumbpctab:tab\X + 4) # tab.stop
0x080008a1

This range should have been 0x08000400 - 0x080008a1; it was offset by 0x400 or 1KiB, exactly the size of the vector table. The stop field was similarly offset by 1KiB. This told me I needed to look in the asmthumbmap function, which was responsible for generating these ranges.

Reading this function a little bit, I saw that the 'L' flag would print the thumb map entries:

tl -T0x08000000 -R0 -L -D0x10000000 \
	-o app/blink/t.out stm32l4/lib/libc.a app/blink/blink.t
tmap: 400-4a1

huh? That looks correct! How does that become 800-8a1 when we read it back in libmach?

while(readn(fd, c, sizeof(c)) == sizeof(c)){
	tab->start = fp->txtaddr + (long)((c[0]<<24)|(c[1]<<16)|(c[2]<<8)|c[3]);
	tab->stop = fp->txtaddr + (long)((c[4]<<24)|(c[5]<<16)|(c[6]<<8)|c[7]);
	tab++;
}

oh! fp->txtaddr is 0x08000400! Reading a.out.h(6), a Plan 9 binary begins with the header:

struct Exec {
	long	magic;	/* magic number */
	long	text;	/* size of text segment */
	long	data;	/* size of initialized data */
	long	bss;	/* size of uninitialized data */
	long	syms;	/* size of symbol table */
	long	entry;	/* entry point */
	long	spsz;	/* size of pc/sp offset table */
	long	pcsz;	/* size of pc/line number table */
};

There is no separate field for the start of the text segment; the entry point is treated as the start of text. The map function in acid, which prints the start, end, and offset (for headers) of each segment, reflects this as well:

% acid /n/dev/mbed9/app/blink/t.out
/n/dev/mbed9/app/blink/t.out:ARM plan 9 boot image
/sys/lib/acid/port
/sys/lib/acid/arm
acid: +map()
{{"text", 0x08000400, 0x080008a8, 0x00000020},
 {"data", 0x08001000, 0x08001000, 0x000004c8}}

I can, as a test, treat the vector table as my entry point:

tl -E vtable -T0x08000000 -R0 -D0x10000000 \
	-o app/blink/t.out stm32l4/lib/libc.a app/blink/blink.t

And now acid computes the correct text addresses, and I can disassemble functions:

% acid app/blink/t.out
app/blink/t.out:ARM plan 9 boot image
/sys/lib/acid/port
/sys/lib/acid/arm
acid: asm(_main)
_main 0x08000400	MOVW	$$setR12,R1
_main+0x2 0x08000402	MOVW	R1,R12
_main+0x4 0x08000404	BL	
_main+0x6 0x08000406	BL	_init
acid: +map()
{{"text", 0x08000000, 0x080004a8, 0x00000020},
 {"data", 0x08001000, 0x08001000, 0x000004c8}}

I discussed this issue with the folks on #cat-v. The issue of the entry point not being the start of the text segment is not something that really happens in ordinary Plan 9 programs, but it does happen with some kernels, which may need to start with vector tables much in the same way the binaries for our stm32 target do. One suggestion made was to modify libmach to mask the lower bytes of the entry point when computing the base address. It would be highly unusual for a binary to not be aligned to the machine's page size (4KiB in this case). So I made the following changes:

There were some fixes necessary after making the changes above; acid will now look for the file /sys/lib/acid/thumb, which had to be created. The direct use of mthumb instead of marm meant the instruction width changed from 4 to 2, which caused PC/line table lookups to fail, as tl(1) unconditionally builds the lookup table with a quanta of 4.

stack traces

The next part is getting a stack trace. The file /sys/lib/acid/$objtype defines an lstk() function which enumerates every call frame on the stack, along with parameter arguments. It looks something like this, for a normal program:

acid: lstk()
await(a0=0x7fffffffeb40)+0xe /sys/src/libc/9syscall/await.s:6
wait()+0x1a /sys/src/libc/9sys/wait.c:13
	buf=0x415448
	fld=0x20e34a
	l=0x20b03d
	w=0x20e779
Waitfor(pid=0x527b)+0x20 /sys/src/cmd/rc/plan9.c:185
	w=0x40ec78
	p=0x420a38
Xsimple()+0x131 /sys/src/cmd/rc/simple.c:103
	a=0x420a38
	pid=0x2044c90000527b
main(argv=0x7fffffffef98,argc=0xfefefefe00000001)+0x32b /sys/src/cmd/rc/exec.c:302
	rcmain=0x4006f0
	num=0xfefe003331313132
	bootstrap=0xfefefefe00000001
	i=0x4006f0fefefefe
_callmain+0x63 /sys/src/libc/9sys/callmain.c:22

And is often the very first thing I'd run in a debugging session. But it doesn't look quite right for our setup:

acid: lstk()
main()+0x2e /n/dev/mbed9/app/blink/blink.c:32
main()+0xfffffffffffffff9 /n/dev/mbed9/libc/stm32l4/l.s:273
/* ... repeat ~40 times ... */
_main+0x1 /n/dev/mbed9/libc/stm32l4/l.s:273

There are about 40 repetitions of the frame

main()+0xfffffffffffffff9 /n/dev/mbed9/libc/stm32l4/l.s:273

which is pointing to the line:

BL	_init(SB)

even though I know that's not true:

acid: line(*PC)
/n/dev/mbed9/app/blink/blink.c:32:	for(i = 0; i < 800000; i++);

The offset main()+0xfffffffffffffff9 looks suspicious, like an integer underflow. The acid functions are more or less sugar around a built-in function, strace, which calls machdata->ctrace. For ARM and Thumb, that is the risctrace function in /sys/src/libmach/machdata.c:368.

int
risctrace(Map *map, uvlong pc, uvlong sp, uvlong link, Tracer trace)
{
	/* ... snip ... */
	while(findsym(pc, CTEXT, &s)) {
		/* ... snip ... */
		if(s.type == 'L' || s.type == 'l' || pc <= s.value+mach->pcquant)
			pc = link;
		/* ... snip ... */
		if(pc == 0 || (pc == oldpc && f.value == 0))
			break;
		/* ... snip ... */
		sp += f.value;
		(*trace)(map, pc-8, sp, &s);

		if(++i > 40)
			break;
	}

The problem here is that if the address in the LINK register is that of a Thumb function, it will always have the LSB set, but that does not mean the instruction is at an unaligned memory location. We can mask that bit off in /sys/lib/acid/thumb:

include("/sys/lib/acid/arm");
defn stk()	{ _stk(*PC, *SP, linkreg(0) & ~1, 0); }
defn lstk()	{ _stk(*PC, *SP, linkreg(0) & ~1, 1); }

Remaining issues

There are a few remaining issues:

However, my current feature set is more than enough to start progressing in this project. I can improve these capabilities over time, as needed.

Part 3 is coming soon!