libpthread, libc, dup2, and crunchgen

I just solved a bug that had been haunting me for a few days.

I've been building a large crunched binary with crunchgen, and ran into a problem with login. I could build the standard binary in /usr/src/usr.bin/login and it ran with no problems, but when I crunched it into my binary, every username I tried to use to login would fail with the error

dup of backchannel: Bad file descriptor

A rather mysterious error. It turns out that it comes from a call to dup2 in libc in /usr/src/lib/libc/gen/auth_sub.c in auth_call, the main call for authenticating logins. I added some code to the kernel to give me better information about why dup2 was failing, and it turned out that sys_dup2 (the system call in kern/kern_descrip.c) was not even getting called! ktrace confirmed this result.

The problem is that dup2 in libc is almost entirely a passthrough to int 0x80 with parameter 90 (the syscall number for dup2), so I was stumped as to how the couple lines of assembly in dup2 could possibly be returning EBADF.

I eventually discovered that libpthread also contains a definition of dup2, and this definition seems to be shadowing the libc definition in the crunched binary. The pthread version, of course, checks some other structures first, and may return EBADF without calling to the kernel.

Indeed, removing the program that needed libpthread (and of course removing the library from linking) solved the problem, and login works again.

The solution, it turns out, is actually trivial. All you need to do is link the offending program that requires pthreads to /usr/lib/libpthread.a directly. Then crunchgen's makefile will call crunchide on the resulting .o file, thus ensuring that the symbols in libpthread.a don't shadow those of libc.