Content tagged c

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System


The game code up until this point abuses timers a lot. It has a timer to handle rendering and to refresh the display, and a timer to change notes of a tune. These tasks are not very time sensitive. A couple of milliseconds delay here or there is not going to be noticeable to users. The timer interrupts are more appropriate for things like maintaining a sound wave of the proper frequency. A slight delay here lowers the quality of the user experience significantly.

We could, of course, do even more complex time management to handle both the graphics and the sound in one loop, but that would be painful. It's much nicer to have a scheduling system that can alternate between multiple threads of execution. It is what I will describe in this post.

Thread Control Block and the Stack

Since there's usually only one CPU, the threads need to share it. The easiest way to achieve time sharing is to have a fixed time slice at the end of which the system will switch to another thread. The systick interrupt perfect for this purpose. Not only is it invoked periodically, but it can also by requested manually by manipulating a register. This property will be useful in implementation of sleeping and blocking.

But first things first: we need to have a structure that will describe a thread, a. k. a. a Thread Control Block:

1struct IO_sys_thread {
2  uint32_t             *stack_ptr;
3  uint32_t              flags;
4  void (*func)();
5  struct IO_sys_thread *next;
6  uint32_t              sleep;
7  IO_sys_semaphore     *blocker;
8  uint8_t               priority;
  • stack_ptr - points to the top of the thread's stack
  • flags - properties describing the thread; we will need just one to indicate whether the thread used the floating-point coprocessor
  • func - thread's entry point
  • next - pointer to the next thread in the queue (used for scheduling)
  • sleep - number of milliseconds the thread still needs to sleep
  • blocker - a pointer to a semaphore blocking the thread (if any)
  • priority - thread's priority

When invoking an interrupt handler, the CPU saves most of the running state of the current thread to the stack. Therefore, the task of the interrupt handler boils down to switching the stack pointers. The CPU will then pop all the registers back from the new stack. This behavior means that we need to do some initialization first:

 1void IO_sys_stack_init(IO_sys_thread *thread, void (*func)(void *), void *arg,
 2  void *stack, uint32_t stack_size)
 4  uint32_t sp1 = (uint32_t)stack;
 5  uint32_t sp2 = (uint32_t)stack;
 6  sp2 += stack_size;
 7  sp2 = (sp2 >> 3) << 3;          // the stack base needs to be 8-aligned
 8  if(sp1 % 4)
 9    sp1 = ((sp1 >> 2) << 2) + 4;  // make the end of the stack 4-aligned
10  stack_size = (sp2 - sp1) / 4;   // new size in double words
12  uint32_t *sp = (uint32_t *)sp1;
13  sp[stack_size-1] = 0x01000000;          // PSR with thumb bit
14  sp[stack_size-2] = (uint32_t)func;      // program counter
15  sp[stack_size-3] = 0xffffffff;          // link register
16  sp[stack_size-8] = (uint32_t)arg;       // r0 - the argument
17  thread->stack_ptr = &sp[stack_size-16]; // top of the stack

The ARM ABI requires that the top of the stack is 8-aligned and we will typically push and pop 4-byte words. The first part of the setup function makes sure that the stack boundaries are right. The second part sets the initial values of the registers. Have a look here for details.

  • the PSR register needs to have the Thumb bit switched on
  • we put the startup function address to the program counter
  • we put 0xffffffff to the link register to avoid confusing stack traces in GDB
  • r0 gets the argument to the startup function
  • an interrupt pushes 16 words worth of registers to the stack, so the initial value of the stack pointer needs to reflect that

This function is typically called as:

1IO_sys_stack_init(thread, thread_wrapper, thread, stack, stack_size);

Note that we do not call the user thread function directly. Rather we have a wrapper function that gets the TBC as its argument. It is because we need to remove the thread from the scheduling queue if the user-specified function returns.

The context switcher

Let's now have a look at the code that does the actual context switching. Since it needs to operate directly on the stack, it needs to be written in assembly. It is not very complicated, though. What it does is:

  • pushing some registers to the stack
  • storing the current stack pointer in the stack_ptr variable of the current TCB
  • calling the scheduler to select the next thread
  • loading the stack pointer from the new thread's TCB
  • popping some registers from the new stack
 1#define OFF_STACK_PTR 0
 2#define OFF_FLAGS     4
 3#define FLAG_FPU      0x01
 5  .thumb
 6  .syntax unified
 8  .global IO_sys_current
 9  .global IO_sys_schedule
11  .text
13  .global systick_handler
14  .type systick_handler STT_FUNC
15  .thumb_func
16  .align  2
18  cpsid i                     ; disable interrupts
19  push  {r4-r11}              ; push r4-11
20  ldr   r0, =IO_sys_current   ; pointer to IO_sys_current to r1
21  ldr   r1, [r0]              ; r1 = OS_current
23  ubfx  r2, lr, #4, #1        ; extract the fourth bit from the lr register
24  cbnz  r2, .Lsave_stack      ; no FPU context to save
25  vstmdb sp!, {s16-s31}       ; push FPU registers, this triggers pushing of
26                              ; s0-s15
27  ldr   r2, [r1, #OFF_FLAGS]  ; load the flags
28  orr   r2, r2, #FLAG_FPU     ; set the FPU context flag
29  str   r2, [r1, #OFF_FLAGS]  ; store the flags
32  str   sp, [r1, #OFF_STACK_PTR] ; store the stack pointer at *OS_current
34  push  {r0, lr}              ; calling c code, so store r0 and the link
35                              ; register
36  bl    IO_sys_schedule       ; call the scheduler
37  pop   {r0, lr}              ; restore r0 and lr
39  ldr   r1, [r0]              ; load the new TCB pointer to r1
40  ldr   sp, [r1, #OFF_STACK_PTR] ; get the stack pointer of the new thread
42  orr   lr, lr, #0x10         ; clear the floating point flag in EXC_RETURN
43  ldr   r2, [r1, #OFF_FLAGS]  ; load the flags
44  tst   r2, #0x01             ; see if we have the FPU context
45  beq   .Lrestore_regs        ; no FPU context
46  vldmia sp!, {s16-s31}       ; pop the FPU registers
47  bic   lr, lr, #0x10         ; set the floating point flag in EXC_RETURN
50  pop   {r4-r11}              ; restore regs r4-11
51  cpsie i                     ; enable interrupts
52  bx    lr                    ;  exit the interrupt, restore r0-r3, r12, lr, pc,
53                              ; psr

The only complication here is that we sometimes need to store the floating point registers in addition to the regular ones. It is, however, only necessary if the thread used the FPU. The fourth bit of EXC_RETURN, the value in the LR register, indicates the status of the FPU. Go here and here for more details. If the value of the bit is 0, we need to save the high floating-point registers to the stack and set the FPU flag in the TCB.

Also, after selecting the new thread, we check if its stack contains the FPU registers by checking the FPU flag in its TCB. If it does, we pop these registers and change EXC_RETURN accordingly.

The Lazy Stacking is taken care of by simply pushing and popping the high registers - it counts as an FPU operation.

Semaphores, sleeping and idling

We can now run threads and switch between them, but it would be useful to be able to put threads to sleep and make them wait for events.

Sleeping is easy. We just need to set the sleep field in the TCB of the current thread and make the scheduler ignore threads whenever their sleep field is not zero:

1void IO_sys_sleep(uint32_t time)
3  IO_sys_current->sleep = time;
4  IO_sys_yield();

The ISR that handles the system time can loop over all threads and decrement this counter every millisecond.

Waiting for a semaphore works in a similar way. We mark the current thread as blocked:

 1void IO_sys_wait(IO_sys_semaphore *sem)
 3  IO_disable_interrupts();
 4  --*sem;
 5  if(*sem < 0) {
 6    IO_sys_current->blocker = sem;
 7    IO_sys_yield();
 8  }
 9  IO_enable_interrupts();

The purpose of IO_sys_yield is to indicate that the current thread does not need to run anymore and force a context switch. The function resets the systick counter and forces the interrupt:

1void IO_sys_yield()
3  STCURRENT_REG = 0;          // clear the systick counter
4  INTCTRL_REG   = 0x04000000; // trigger systick

Waking a thread waiting for a semaphore is somewhat more complex:

 1void IO_sys_signal(IO_sys_semaphore *sem)
 3  IO_disable_interrupts();
 4  ++*sem;
 5  if(*sem <= 0 && threads) {
 6    IO_sys_thread *t;
 7    for(t = threads; t->blocker != sem; t = t->next);
 8    t->blocker = 0;
 9  }
10  IO_enable_interrupts();

If the value of the semaphore was negative, we find a thread that it was blocking and unblock it. It will make the scheduler consider this thread for running in the future.

None of the user-defined threads may be runnable at the time the scheduler makes its decision. All of them may be either sleeping or waiting for a semaphore. In that case, we need to keep the CPU occupied with something, i.e., we need a fake thread:

1static void iddle_thread_func(void *arg)
3  (void)arg;
4  while(1) IO_wait_for_interrupt();


The system maintains a circular linked list of TCBs called threads. The job of the scheduler is to loop over this list and select the next thread to run. It places its selection in a global variable called IO_sys_current so that other functions may access it.

 1void IO_sys_schedule()
 3  if(!threads) {
 4    IO_sys_current = &iddle_thread;
 5    return;
 6  }
 8  IO_sys_thread *stop = IO_sys_current->next;
10  if(IO_sys_current == &iddle_thread)
11    stop = threads;
13  IO_sys_thread *cur  = stop;
14  IO_sys_thread *sel  = 0;
15  int            prio = 266;
17  do {
18    if(!cur->sleep && !cur->blocker && cur->priority < prio) {
19      sel = cur;
20      prio = sel->priority;
21    }
22    cur = cur->next;
23  }
24  while(cur != stop);
26  if(!sel)
27    sel = &iddle_thread;
29  IO_sys_current = sel;

This scheduler is simple:

  • whenever there is nothing to run, select the idle thread
  • otherwise select the next highest priority thread that is neither sleeping nor blocked on a semaphore

Starting up the beast

So how do we get this whole business running? We need to invoke the scheduler that will preempt the current thread and select the next one to run. The problem is that we're running using the stack provided by the bootstrap code and don't have a TCB. Nothing prevents us from creating a dummy one, though. We can create it on the current stack (it's useful only once) and point it to the beginning of our real queue of TCBs:

1IO_sys_thread dummy; = threads;
3IO_sys_current = &dummy;

We then set the systick up:

1STCTRL_REG     = 0;            // turn off
2STCURRENT_REG  = 0;            // reset
3SYSPRI3_REG   |= 0xE0000000;   // priority 7
4STRELOAD_REG   = time_slice-1; // reload value

And force its interrupt:



Tests 11 and 12 run a dummy calculation for some time and then return. After this happens, the system can only run the idle thread. If we plug-in the profiler code, we can observe the timings on a logic analyzer:

Test #11
Test #11

Test 13 is more complicated than the two previous ones. Three threads are running in a loop, sleeping, and signaling semaphores. Two more threads are waiting for these semaphores, changing some local variables and signaling other semaphores. Finally, there is the writer thread that blocks on the last set of semaphores and displays the current state of the environment. The output from the logic analyzer shows that the writer thread needs around 3.3 time-slices to refresh the screen:

Test #13
Test #13

Silly Invaders

How all this makes Silly Invaders better? The main advantage is that we don't need to calculate complex timings for multiple functions of the program. We create two threads, one for rendering of the scene and another one for playing the music tune. Each thread cares about its own timing. Everything else takes care of itself with good enough time guarantees.

 1IO_sys_thread game_thread;
 2void game_thread_func()
 4  while(1) {
 5    SI_scene_render(&scenes[current_scene].scene, &display);
 6    IO_sys_sleep(1000/scenes[current_scene].scene.fps);
 7  }
10IO_sys_thread sound_thread;
11void sound_thread_func()
13  IO_sound_player_run(&sound_player);

The threads are registered with the system in the main function:

1IO_sys_thread_add(&game_thread,  game_thread_func,  2000, 255);
2IO_sys_thread_add(&sound_thread, sound_thread_func, 1000, 255);

For the complete code see:


There is a great course on EdX called Realtime Bluetooth Networks explaining the topic in more details. I highly recommend it.


It took some playing around, but I have finally managed to figure out how to build from source all the tools necessary to put Zephyr on Arduino 101. You may say that the effort is pointless because you could just use whatever is provided by the SDK. For me, however, the deal is more about what I can learn from the experience that about the result itself. There is enough open source code around to make things work reasonably well, but putting it all together is a bit of a challenge, so what follows is a short HOWTO.

Arduino 101 setup
Arduino 101 setup


Arduino 101 has a Quark core and an ARC EM core. The appropriate targets seem to be i586-none-elfiamcu and arc-none-elf for the former and the later respectively. Since there is no pre-packaged toolchain for either of these in Debian, you'll need to build your own. You can use the vanilla binutils (version 2.27 worked for me) and the vanilla newlib (version did not require any patches). GCC is somewhat more problematic. Since apparently not all the necessary ARC patches have been accepted into the mainline yet, you'll need to download it from the Synopsys GitHub repo. GDB requires tweaking for both cores.


]==> mkdir binutils && cd binutils
]==> wget
]==> tar jxf binutils-2.27.tar.bz2
]==> mkdir i586-none-elfiamcu && cd i586-none-elfiamcu
]==> ../binutils-2.27/configure --prefix=/home/ljanyst/Apps/cross-compilers/i586-none-elfiamcu --target=i586-none-elfiamcu
]==> make -j12 && make install
]==> cd .. && mkdir arc-none-elf && arc-none-elf
]==> ../binutils-2.27/configure --prefix=/home/ljanyst/Apps/cross-compilers/arc-none-elf --target=arc-none-elf
]==> make -j12 && make install
]==> cd ../..


]==> mkdir gcc && cd gcc
]==> wget
]==> tar jxf gcc-6.2.0.tar.bz2
]==> git clone
]==> cd gcc && git checkout arc-4.8-dev && cd ..
]==> mkdir i586-none-elfiamcu && cd i586-none-elfiamcu
]==> ../gcc-6.2.0/configure --prefix=/home/ljanyst/Apps/cross-compilers/i586-none-elfiamcu --target=i586-none-elfiamcu --enable-languages=c --with-newlib
]==> make -j12 all-gcc && make install-gcc
]==> cd .. && mkdir arc-none-elf && arc-none-elf
]==> ../gcc/configure --prefix=/home/ljanyst/Apps/cross-compilers/arc-none-elf --target=arc-none-elf  --enable-languages=c --with-newlib --with-cpu=arc700
]==> make -j12 all-gcc && make install-gcc
]==> cd ../..


]==> mkdir newlib && cd newlib
]==> wget
]==> tar zxf newlib-
]==> mkdir i586-none-elfiamcu && cd i586-none-elfiamcu
]==> ../newlib- --prefix=/home/ljanyst/Apps/cross-compilers/i586-none-elfiamcu --target=i586-none-elfiamcu
]==> make -j12 && make install
]==> cd .. && mkdir arc-none-elf && arc-none-elf
]==> ../newlib- --prefix=/home/ljanyst/Apps/cross-compilers/arc-none-elf --target=arc-none-elf
]==> make -j12 && make install
]==> cd ../..


]==> cd gcc/i586-none-elfiamcu
]==> make -j12 all-target-libgcc && make install-target-libgcc
]==> cd ../arc-none-elf
]==> make -j12 all-target-libgcc && make install-target-libgcc
]==> cd ../..

GDB does not work for either platform out of the box. For Quark it compiles the i386 version but does not recognize the iamcu architecture even though, according to Wikipedia, it's essentially the same as i586 and libbfd knows about it. After some poking around the code, it seems that initilizing the i386 platform with iamcu bfd architecture definition does the trick:

 1diff -Naur gdb-7.11.1.orig/gdb/i386-tdep.c gdb-7.11.1/gdb/i386-tdep.c
 2--- gdb-7.11.1.orig/gdb/i386-tdep.c     2016-06-01 02:36:15.000000000 +0200
 3+++ gdb-7.11.1/gdb/i386-tdep.c  2016-09-24 15:39:11.000000000 +0200
 4@@ -8890,6 +8890,7 @@
 5 _initialize_i386_tdep (void)
 6 {
 7   register_gdbarch_init (bfd_arch_i386, i386_gdbarch_init);
 8+  register_gdbarch_init (bfd_arch_iamcu, i386_gdbarch_init);
10   /* Add the variable that controls the disassembly flavor.  */
11   add_setshow_enum_cmd ("disassembly-flavor", no_class, valid_flavors,

For ARC the Synopsys open source repo provides a solution.

]==> mkdir gdb
]==> wget
]==> tar xf gdb-7.11.1.tar.xz
]==> cd gdb-7.11.1 && patch -Np1 -i ../iamcu-tdep.patch && cd ..
]==> git clone
]=> cd binutils-gdb && git checkout arc-2016.09-gdb && cd ..
]==> mkdir i586-none-elfiamcu && cd i586-none-elfiamcu
]==> ../gdb-7.11.1/configure --prefix=/home/ljanyst/Apps/cross-compilers/i586-none-elfiamcu --target=i586-none-elfiamcu
]==> make -j12 && make install
]==> cd .. && mkdir arc-none-elf && arc-none-elf
]==> ../binutils-gdb/configure --prefix=/home/ljanyst/Apps/cross-compilers/arc-none-elf --target=arc-none-elf 
]==> make -j12 all-gdb && make install-gdb
]==> ../..


There was no OpenOCD release for quite some time, and it does not seem to have any support for Quark SE. The situation is not much better if you look at the head of the master branch of their repo. Fortunately, both Intel and Synopsys provide some support for their parts of the platform and making it work with mainline openocd does not seem to be hard.

]==> git clone && cd openocd
]==> git checkout lj
]==> ./bootstrap
]==> ./configure --prefix=/home/ljanyst/Apps/openocd
]==> make -j12 && make install

Zephyr uses the following configuration for the Arduino (referred to as openocd.conf below):

 1source [find interface/ftdi/flyswatter2.cfg]
 2source [find board/quark_se.cfg]
 4quark_se.quark configure -event gdb-attach {
 5        reset halt
 6        gdb_breakpoint_override hard
 9quark_se.quark configure -event gdb-detach {
10        resume
11        shutdown

You can use the following commands to run the GDB server, flash for Quark and flash for ARC respectively (this is what Zephyr does):

]==> openocd -s /home/ljanyst/Apps/openocd/share/openocd/scripts/ -f openocd.cfg  -c 'init' -c 'targets' -c 'reset halt'
]==> openocd -s /home/ljanyst/Apps/openocd/share/openocd/scripts/ -f openocd.cfg  -c 'init' -c 'targets' -c 'targets quark_se.arc-em' -c 'reset halt' -c 'load_image zephyr.bin 0x40010000' -c 'reset halt' -c 'verify_image zephyr.bin 0x40010000' -c 'reset run' -c 'shutdown'
]==> openocd -s /home/ljanyst/Apps/openocd/share/openocd/scripts/ -f openocd.cfg  -c 'init' -c 'targets' -c 'targets quark_se.arc-em' -c 'reset halt' -c 'load_image zephyr.bin 0x40034000' -c 'reset halt' -c 'verify_image zephyr.bin 0x40034000' -c 'reset run' -c 'shutdown'

Hello world!

You need to compile and flash Zephyr's Hello World sample. The two commands below do the trick for the compilation part:

make BOARD=arduino_101_factory CROSS_COMPILE=i586-none-elfiamcu- CFLAGS="-march=lakemont -mtune=lakemont -msoft-float -miamcu -O0"
make BOARD=arduino_101_sss_factory CROSS_COMPILE=arc-none-elf-

After flashing, you should see the following on your UART console:

]==> screen /dev/ttyUSB0 115200,cs8
ipm_console0: 'Hello World! arc'
Hello World! x86


If you follow the instructions from the Zephyr wiki, debugging for the Intel part works fine. I still have some trouble making breakpoints work for ARC and will try to write an update if I have time to figure it out.

]==> i586-none-elfiamcu-gdb outdir/zephyr.elf
(gdb) target remote :3333
Remote debugging using :3333
0x0000fff0 in ?? ()
(gdb) b main
Breakpoint 1 at 0x400100ed: file /home/ljanyst/Projects/zephyr/zephyr-project/samples/hello_world/nanokernel/src/main.c, line 37.
(gdb) c
target running
hit hardware breakpoint (hwreg=0) at 0x400100ed

Breakpoint 1, main () at /home/ljanyst/Projects/zephyr/zephyr-project/samples/hello_world/nanokernel/src/main.c:37
37              PRINT("Hello World! %s\n", CONFIG_ARCH);
(gdb) s
step done from EIP 0x400100ed to 0x400100f2
step done from EIP 0x400100f2 to 0x400100f7
step done from EIP 0x400100f7 to 0x40013129
target running
hit hardware breakpoint (hwreg=1) at 0x4001312f
printk (fmt=0x40013e04 "Hello World! %s\n") at /home/ljanyst/Projects/zephyr/zephyr-project/misc/printk.c:164
164             va_start(ap, fmt);
(gdb) s
step done from EIP 0x4001312f to 0x40013132
step done from EIP 0x40013132 to 0x40013135
165             _vprintk(fmt, ap);


My medium-term goal is to port my Silly Invaders game to a Real Time Operating System. Zephyr seems to be a good choice. It's open source, operates under the auspices of the Linux Foundation and has an active community with many developers from Intel committing the code.

They, unfortunately, do not support Tiva so I will need to port the OS before I can proceed with the application. I decided to buy the Freescale K64F board, which is supported, to familiarize myself a little with Zephyr before I start the porting work. The howto page for setting up K64F seems to be terribly complicated and requires a JTAG programmer. I summarize here a simpler way using cmsis-dap over USB.


I updated the MBED interface firmware following the instructions on this site. I also build my own OpenOCD from the head of the master branch using the following configuration options:

./configure --prefix=/home/ljanyst/Apps/openocd  --enable-cmsis-dap

Things may work fine with the stock firmware and the stock OpenOCD as well, but I did not try that. It's also probably a good idea to add the following udev rule so that you don't have to run things as root:

]==> cat /etc/udev/rules.d/99-openocd.rules
# frdm-k64f
ATTRS{idVendor}=="0d28", ATTRS{idProduct}=="0204", GROUP="plugdev", MODE="0660"
]==> sudo udevadm control --reload-rules

Hello world!

I use the ARM cross-compiler provided by Debian to compile Zephyr and then just copy the resulting binary to the MBED disk:

]==> cd samples/hello_world/nanokernel
]==> make BOARD=frdm_k64f CROSS_COMPILE=arm-none-eabi- CFLAGS=-O0
]==> cp outdir/zephyr.bin /media/ljanyst/MBED/

You can see the effects in the UART console using screen:

]==> screen /dev/ttyACM0 115200,cs8
Hello World!

I then run OpenOCD using the following script:

]==> cat k64f.cfg
set CHIPNAME k60
source [find target/kx.cfg]

$_TARGETNAME configure -event gdb-attach {

]==> openocd -s /home/ljanyst/Apps/openocd/share/openocd/scripts/ -c "interface cmsis-dap" -f k64f.cfg

And GDB:

]==> cat remote1.conf
target extended-remote :3333
monitor reset init
break main
]==> arm-none-eabi-gdb  --command=remote1.conf outdir/zephyr.elf
Breakpoint 1 at 0x129c: file /home/ljanyst/Projects/zephyr/zephyr-project/samples/hello_world/nanokernel/src/main.c, line 37.
Note: automatically using hardware breakpoints for read-only addresses.

Breakpoint 1, main () at /home/ljanyst/Projects/zephyr/zephyr-project/samples/hello_world/nanokernel/src/main.c:37
37              PRINT("Hello World!\n");
(gdb) s
printk (fmt=0x2c90 "Hello World!\n") at /home/ljanyst/Projects/zephyr/zephyr-project/misc/printk.c:164
164             va_start(ap, fmt);
(gdb) s
165             _vprintk(fmt, ap);
(gdb) s
_vprintk (fmt=0x2c90 "Hello World!\n", ap=...) at /home/ljanyst/Projects/zephyr/zephyr-project/misc/printk.c:75
75              int might_format = 0; /* 1 if encountered a '%' */
(gdb) where
#0  _vprintk (fmt=0x2c90 "Hello World!\n", ap=...) at /home/ljanyst/Projects/zephyr/zephyr-project/misc/printk.c:75
#1  0x00001b46 in printk (fmt=0x2c90 "Hello World!\n") at /home/ljanyst/Projects/zephyr/zephyr-project/misc/printk.c:165
#2  0x000012a2 in main () at /home/ljanyst/Projects/zephyr/zephyr-project/samples/hello_world/nanokernel/src/main.c:37

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System

Random Number Generator

To make the game more engaging, we introduce some randomness into it. We don't need anything cryptographically secure, so a Linear Congruential Generator will do just fine. We count the time from the start-up in millisecond-long jiffies and wait for a first button press to select the seed.

 1void button_event(IO_io *io, uint16_t event)
 3  uint64_t btn;
 4  IO_get(io, &btn);
 5  if(btn)
 6    button_value = 1;
 8  if(!rng_initialized) {
 9    rng_initialized = 1;
10    IO_rng_seed(IO_time());
11  }

Rendering Engine

The rendering engine takes a scene descriptor, a display device, and a timer. Based on this information it computes new positions of objects, draws them on the screen if necessary and checks for collisions.

 1struct SI_scene {
 2  SI_object **objects;
 3  void      (*pre_render)(struct SI_scene *);
 4  void      (*collision)(SI_object *obj1, SI_object *obj2);
 5  uint8_t     fps;
 6  uint8_t     num_objects;
 7  uint8_t     flags;
10void SI_scene_render(SI_scene *scene, IO_io *display, IO_io *timer);

Each SI_scene holds a list of "polymorphic" objects that should be rendered, a pointer to a pre_render function that calculates a new position of each object, and a pointer to a collision callback that is invoked when the scene renderer detects an overlap between two objects. The SI_scene_render function runs after every interrupt:

1  while(1) {
2    SI_scene_render(&scenes[current_scene].scene, &display, &scene_timer);
3    IO_wait_for_interrupt();
4  }

Whether it gets executed or not, depends on the flag parameter of the scene. If it's set to SI_SCENE_IGNORE, the renderer returns immediately. On the other hand, if it's set to SI_SCENE_RENDER, the renderer calls the pre_render callback, draws the objects on the screen, and computes the object overlaps notifying the collision callback if necessary. After each frame, the scene is disabled (SI_SCENE_IGNORE). It is re-enabled by the timer interrupt in a time quantum that depends on the fps parameter.

See SI_scene.h and SI_scene.c.

Each object has a draw function that enables the renderer to draw it on the screen. There are three types of objects: a generic object, a bitmap object, and a text object:

 1struct SI_object {
 2  uint16_t x;
 3  uint16_t y;
 4  uint16_t width;
 5  uint16_t height;
 6  uint8_t  flags;
 7  uint8_t  user_flags;
 8  void (*draw)(struct SI_object *this, IO_io *display);
11struct SI_object_bitmap {
12  SI_object        obj;
13  const IO_bitmap *bmp;
16struct SI_object_text {
17  SI_object      obj;
18  const char    *text;
19  const IO_font *font;

The object array in the scene is initialized with the SI_object pointers:

1static SI_object         score_obj;
2static SI_object_bitmap  invader_obj[5];
3scene->objects[1] = &score_obj;
4scene->objects[i+5] = &invader_obj[i].obj;

See SI_scene_game.c.

The renderer calls the draw function of each SI_OBJECT_VISIBLE object:

1obj->draw(obj, display);

Finally, each draw method uses the CONTAINER_OF macro to compute the pointer to the actual object of concrete type:

 2  ((TYPE *) ( (char *)MEMBER_ADDR - offsetof(TYPE, MEMBER)))
 4void SI_object_bitmap_draw(SI_object *obj, IO_io *display)
 6  SI_object_bitmap *this = CONTAINER_OF(SI_object_bitmap, obj, obj);
 7  IO_display_print_bitmap(display, obj->x, obj->y, this->bmp);
10void SI_object_text_draw(SI_object *obj, IO_io *display)
12  SI_object_text *this = CONTAINER_OF(SI_object_text, obj, obj);
13  IO_display_set_font(display, this->font);
14  IO_display_cursor_goto(display, obj->x, obj->y);
15  IO_print(display, "%s", this->text);

The Game

All this seems to work pretty well when put together:

The Game

See silly-invaders.c.

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System


Tiva does not have a DAC, but we'd like to have some sound effects while playing the game. Fortunately, it's easy to make a simple binary-weighted DAC using resistors and GPIO signals. It's not very accurate, but will do.

A binary-weighted DAC
A binary-weighted DAC

As far as the software is concerned, we will simply take 4 GPIO pins and set them up as output. We will then get an appropriate bit-banded alias such that writing an integer to it is reflected only in the state of these four pins.

 1int32_t IO_dac_init(IO_io *io, uint8_t module)
 3  if(module > 0)
 4    return -IO_EINVAL;
 6  TM4C_gpio_port_init(GPIO_PORTD_NUM);
 7  TM4C_gpio_pin_init(GPIO_PORTD_NUM, GPIO_PIN0_NUM, 0, 0, 1);
 8  TM4C_gpio_pin_init(GPIO_PORTD_NUM, GPIO_PIN1_NUM, 0, 0, 1);
 9  TM4C_gpio_pin_init(GPIO_PORTD_NUM, GPIO_PIN2_NUM, 0, 0, 1);
10  TM4C_gpio_pin_init(GPIO_PORTD_NUM, GPIO_PIN3_NUM, 0, 0, 1);
12  uint32_t addr =  GPIO_REG_BASE + GPIO_PORTD;
13  addr += GPIO_PIN0_BIT_OFFSET;
14  addr += GPIO_PIN1_BIT_OFFSET;
15  addr += GPIO_PIN2_BIT_OFFSET;
16  addr += GPIO_PIN3_BIT_OFFSET;
17  dac_data = (uint32_t*)addr;
19  io->type    = IO_DAC;
20  io->sync    = 0;
21  io->channel = 0;
22  io->flags   = 0;
23  io->read    = 0;
24  io->write   = dac_write;
25  io->event   = 0;
27  return 0;

See TM4C_platform01.c.


We will create a virtual device consisting of a DAC and a timer. Using the timer, we will change the output of the DAC frequently enough to produce sound. Since the timer interrupt needs to be executed often and any delay makes the sound break, we need to assign the highest possible priority to this interrupt so that it does not get preempted.

1int32_t IO_sound_init(IO_io *io, uint8_t module)
4  IO_dac_init(&snd_dac, 0);
5  IO_timer_init(&snd_timer, 11);
6  TM4C_enable_interrupt(104, 0); // adjust the interrupt priority for timer 11
7  snd_timer.event = snd_timer_event;

Writing to this virtual device sets the frequency of the tone that we want to play by adjusting the timer's firing rate accordingly.

 1static int32_t snd_write(IO_io *io, const void *data, uint32_t length)
 3  if(length != 1)
 4    return -IO_EINVAL;
 5  const uint64_t *val = data;
 7  if(!(*val)) {
 8    snd_interval = 0;
 9    return 1;
10  }
11  uint8_t turn_on = 0;
12  if(!snd_interval)
13    turn_on = 1;
15  double interval = 1.0/(*val);
16  interval /= 32.0;
17  interval *= 1000000000;
18  snd_interval = interval;
20  if(turn_on)
21    IO_set(&snd_timer, interval);
22  return 1;

In reality, the timer fires 32 times more often than the frequency of the tone requires. It is because we use a table with 32 entries to simulate the actual sound wave. In principle, we could just use a sinusoid, but it turns out that the quality of the sound is not so great if we do so. I have found another waveform in the lab materials of EdX's Embedded Systems course that works much better.

 1static const uint8_t snd_trumpet[] = {
 2  10, 11, 11, 12, 10,  8,  3,  1,  8, 15, 15, 11, 10, 10, 11, 10, 10, 10, 10,
 3  10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 10, 10, 10 };
 5static void snd_timer_event(IO_io *io, uint16_t event)
 7  IO_set(&snd_dac, snd_trumpet[snd_step++]);
 8  snd_step %= 32;
 9  if(snd_interval)
10    IO_set(io, snd_interval);

See TM4C_platform01.c.

Nokia tunes

The tune player API consists of four functions:

1int32_t IO_sound_play(IO_io *io, IO_io *timer, IO_tune *tune, uint16_t start);
2int32_t IO_sound_stop(IO_io *io);
3IO_tune *IO_sound_decode_RTTTL(const char *tune);
4void IO_sound_free_tune(IO_tune *tune);
  • IO_sound_play uses a sound device and a timer to play a tune. It sends an IO_EVENT_DONE to the virtual sound device when the playback finishes.
  • IO_sound_stop stops the playback on the given device and returns the index of the last note it played so that it can be restarted from that point.
  • IO_sound_decode_RTTL takes an RTTTL representation and produces the IO_tune structure that can be handled by the player.
  • IO_sound_free_tune frees the memory used by IO_sound_decode_RTTL when it's no longer needed.

There is plenty of tunes all over the Internet. The ones in the demo video are taken from here. The code of the player is based on this work.

It plays music! :)

See IO_sound.c.

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System


Tiva has 12 timer modules that can be configured in various relatively complex ways. However, for the purpose of this game, we don't need anything fancy. We will, therefore, represent a timer as an IO_io device with the IO_set function (generalized from IO_gpio_set) setting and arming it. When it counts to 0, the IO_TICK event will be reported to the event handler.

 1void timer_event(IO_io *io, uint16_t event)
 4  IO_set(&timer, 500000000); // fire in half second
 7int main()
 9  IO_init();
10  IO_timer_init(&timer, 0);
11  timer.event = timer_event;
12  IO_set(&timer, 500000000); // fire in half second
14  while(1)
15    IO_wait_for_interrupt();

See TM4C_timer.c and test-07-timer.c.


Similarly to the timers, the ADC sequencers on Tiva may be set up in fairly sophisticated ways. There are 12 analog pins, two modules with four sequencers each. Again, we don't need anything sophisticated here, so we will just use the first eight pins and assign them to a separate sequencer each. In the blocking mode, IO_get initiates the readout and returns the value. In the non-blocking and asynchronous mode IO_set, requests sampling and IO_get returns it when ready. An IO_DONE event is reported to the event handler if enabled.

 1IO_io slider;
 2IO_io timer;
 3uint64_t sliderR = 0;
 5void timer_event(IO_io *io, uint16_t event)
 7  IO_set(&slider, 0); // request a sample
10void slider_event(IO_io *io, uint16_t event)
12  IO_get(&slider, &sliderR);
13  IO_set(&timer, 100000000); // fire in 0.1 sec
16int main()
18  IO_init();
20  IO_timer_init(&timer, 0);
21  IO_slider_init(&slider, 0, IO_ASYNC);
23  timer.event     = timer_event;
24  slider.event    = slider_event;
26  IO_event_enable(&slider,    IO_EVENT_DONE);
28  IO_set(&slider, 0); // request a sample
30  while(1)
31    IO_wait_for_interrupt();

See TM4C_adc.c and test-08-input.c.

The game board

Everything works fine when soldered together as well.

Buttons and the Slider

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System


To test and debug the SSI code, I connected two boards and made them talk to each other. It mostly worked. However, it turned out that, by default, you can run the OpenOCD-GDB duo only for one board at a time. It's the one that libusb enumerates first. There is a patch that lets OpenOCD choose the device to attach to by its serial number. The patch has not made it to any release yet, but applying it and recompiling the whole thing is relatively straight-forward: clone the source, apply the patch and run the usual autotools combo. You will then need to create a config file for each board that specifies unique port numbers and defines the serial number of the device to attach to:

]==> cat board1.cfg
gdb_port 3333
telnet_port 4444
tcl_port 6666
interface hla
hla_serial 0Exxxxxx
source [find board/ek-tm4c123gxl.cfg]

]==> cat board2.cfg
gdb_port 3334
telnet_port 4445
tcl_port 6667
interface hla
hla_serial 0Exxxxxx
source [find board/ek-tm4c123gxl.cfg]

Separate GDB batch files come handy as well:

]==> cat gdb-board1.conf
target extended-remote :3333
monitor reset init
break main

]==> cat gdb-board2.conf
target extended-remote :3334
monitor reset init
break main

Tweaking the linker script

GCC started generating .init and .fini sections that contain no-op functions:

]==> arm-none-eabi-objdump -d  test-06-display.axf
Disassembly of section .init:

00007af8 <_init>:
    7af8:       b5f8            push    {r3, r4, r5, r6, r7, lr}
    7afa:       bf00            nop

Disassembly of section .fini:

00007afc <_fini>:
    7afc:       b5f8            push    {r3, r4, r5, r6, r7, lr}
    7afe:       bf00            nop

We will discard this code by adding the following to the linker script:

1  /DISCARD/ :
2  {
3    *(.init*)
4    *(.fini*)
5  }

GCC also started generating the stack unwinding code and GDB gets confused in some places if it is not present, so we put it after the code in FLASH:

1  .ARM.exidx :
2  {
3    *(.ARM.exidx*)
4    *(.gnu.linkonce.armexidx*)
5  } > FLASH

See TM4C.ld.


We need both SSI and GPIO to control the Nokia display that we want to use for the game. Since, in the end, both these systems need to push and receive data, they fit well the generic interface used for UART. The SSI's initialization function needs many more parameters than the one for UART, so we pack them all in a struct. As far as GPIO is concerned, there are two helpers: IO_gpio_get_state and IO_gpio_set_state that just write the appropriate byte to the IO device. GPIO also comes with a new event type: IO_EVENT_CHANGE.

1struct IO_ssi_attrs {
2  uint8_t  master;        //!< 1 for master, 0 for slave
3  uint8_t  slave_out;     //!< 1 slave output enabled, 0 slave output disabled
4  uint32_t bandwidth;     //!< bandwidth in bps
5  uint8_t  frame_format;  //!< frame format
6  uint8_t  freescale_spo; //!< SPO value for freescale frames
7  uint8_t  freescale_sph; //!< SPH value for freescale frames
8  uint8_t  frame_size;    //!< size of the frame in bits

See TM4C_ssi.c and TM4C_gpio.c.

Platforms, the display interface, and fonts

All the devices that are not directly on the board may be connected in many different ways. To handle all these configurations with the same board, we split the driver into libtm4c.a (for the board specific stuff) and libtm4c_platform_01.a (for the particular configuration). For now, the only thing that the platform implements is the display interface. It passes the appropriate SSI module and GPIOs to the actual display driver. The user sees the usual IO_io structure that is initialized with IO_display_init and can be written to and synced. write renders the text to the back-buffer, while sync sends the back-buffer to the device for rendering. There's also a couple of specialized functions that have to do only with display devices:

 1int32_t IO_display_get_attrs(IO_io *io, IO_display_attrs *attrs);
 2int32_t IO_display_clear(IO_io *io);
 3int32_t IO_display_put_pixel(IO_io *io, uint16_t x, uint16_t y, uint32_t argb);
 4int32_t IO_display_print_bitmap(IO_io *io, uint16_t x, uint16_t y,
 5  const IO_bitmap *bitmap);
 6int32_t IO_display_set_font(IO_io *io, const IO_font *font);
 7int32_t IO_display_cursor_goto(IO_io *io, uint32_t x, uint32_t y);
 8int32_t IO_display_cursor_goto_text(IO_io *io, uint32_t line, uint32_t space);
 9int32_t IO_display_cursor_move(IO_io *io, int32_t dx, int32_t dy);
10int32_t IO_display_cursor_move_text(IO_io *io, int32_t dline, int32_t dspace);

See IO_display.h.

Platform 01 provides one display device, a PCD8544, the one used in Nokia 5110. It translates and passes the interface calls to the lower-level driver. See pcd8544.c.

If you haven't noticed in the list of the functions above, the display interface supports multiple fonts. In fact, I wrote a script that rasterizes TrueType fonts and creates internal IO_font structures. These can then be used to render text on a display device. All you need to do is provide a TTF file, declare the font name and size in CMake, and then reference it in the font manager. The code comes with DejaVuSans10 and DejaVuSerif10 by default.

The heap

Malloc comes handy from time to time, so I decided to implement one. It is extremely prone to fragmentation and never merges chunks, so using free is not advisable. Still, sometimes you just wish you had one. For instance, when you need to define a buffer for pixels and don't have a good way to ask for display parameters at compile time. For alignment reasons, the heap management code reserves a bit more than 4K for the stack. It then creates a 32 bytes long guard region protected by the MPU. Everything between the end of the .bss section and the guard page is handled by IO_malloc and IO_free.

 1void TM4C_heap_init()
 3  uint8_t *stack_start = (uint8_t *)0x20007ff8;
 4  uint8_t *stack_end   = stack_start-4120;
 5  uint8_t *stack_guard = stack_end-32;
 6  uint8_t *heap_start  = (uint8_t *)&__bss_end_vma;
 8  MPUCTRL_REG |= (uint32_t)0x05; // enable MPU and the background region
10  uint32_t val = (uint32_t)stack_guard;
11  val |= 0x10; // valid
12  val |= 0x07; // highest priority region
13  MPUBASE_REG &= ~0xfffffff7;
14  MPUBASE_REG |= val;
16  val = 0;
17  val |= (1 << 28); // disable instruction fetches
18  val |= (4 << 1);  // 0x04 == 32bytes
19  val |= 1;         // enable the region
20  MPUATTR_REG &= ~0x173fff3f;
21  MPUATTR_REG |= val;
23  IO_set_up_heap(heap_start, stack_guard);

See TM4C.c and IO_malloc.c.

A display test

The LCD demo works fine on the breadboard. As you can see, there is a text printed with two kinds of fonts: with and without serifs. Later, the code plays the Game of Life shooting gliders.

Glider Gun on a breadboard


Since the display works fine, it's safe to do some soldering. We'll use a 9-volt battery as a power source and an LM1086 power regulator to supply 3.3 volts to the microcontroller and other devices.

Soldered Display - Front
Soldered Display - Front

Soldered Display - Back
Soldered Display - Back

Glider Gun - Soldered

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System

Hardware Abstraction Layer

I'd like the game to be as portable as possible. As far as the game logic is concerned, the actual interaction with the hardware is immaterial. Ideally, we just need means to write a pixel to a screen, blink an LED or check the state of a push-button. It means that hiding the hardware details behind a generic interface is desirable. This interface can then be re-implemented for a different kind of board, and the whole thing can act as a cool tool for getting to know new hardware.

In this project, we will use one static library (libio.a) to provide the interface. This library will implement all the hardware independent functions as well as the stubs for the driver (as weak symbols). Another library (libtm4c.a) will provide the real driver logic for Tiva and the strong symbols. This kind of approach enables us to use the linker to easily produce the final binary for other platforms in the future.

Initialization PLL and FPU

To initialize the hardware platform, the user calls IO_init(). The stub for this function is provided by libio.a as follows:

1int32_t __IO_init()
3  return -IO_ENOSYS;
6WEAK_ALIAS(__IO_init, IO_init);

The actual implementation for Tiva in libtm4c.a initializes PLL to provide 80MHz clock and turns on microDMA. It also sets the access permissions to the FPU by setting the appropriate bits in the CPAC register and resetting the pipeline in assembly. We will likely need the floating point in the game, and it comes handy when calculating UART transmission speed parameters.

 1int32_t IO_init()
 3  TM4C_pll_init();
 4  TM4C_dma_init();
 6  // Enable the floating point coprocessor
 7  CPAC_REG |= (0x0f << 20);
 8  __asm__ volatile (
 9    "dsb\r\n"        // force memory writed before continuing
10    "isb\r\n" );     // reset the pipeline
11  return 0;

Simple read/write interface and functions

We provide an IO device abstraction called IO_io and implement four generic functions for accessing it:

1int32_t IO_write(IO_io *io, const void *data, uint32_t length);
2int32_t IO_print(IO_io *io, const char *format, ...);
3int32_t IO_read(IO_io *io, void *data, uint32_t length);
4int32_t IO_scan(IO_io *io, uint8_t type, void *data, uint32_t param);

IO_read and IO_write push to and fetch bytes from the device. IO_print writes a formated string to the device using the standard printf semantics. IO_scan reads a word (a stream of characters surrounded by whitespaces) and tries to convert it to the requested type.

Each subsystem needs to provide its initialization function to fill the IO_io struct with the information required to perform the IO operations. For instance, the following function initializes UART:

1int32_t IO_uart_init(IO_io *io, uint8_t module, uint16_t flags, uint32_t baud);

It needs to know which UART module to use, what the desired mode of operation is (non-blocking, asynchronous, DMA...) and what should be the speed of the link. This approach hides the hardware details from the user well and is very generic, see test-01-uart.c. For instance, you can write something like this:

2IO_io uart0;
3IO_uart_init(&uart0, 0, 0, 115200);
4IO_print(&uart0, "Hello %s\r\n", "World");

Passing 0 as flags to the UART initialization routine creates a blocking device that is required for IO_print and IO_scan to work.

Non-blocking and asynchronous IO

A blocking IO device will cause the IO functions to return only after they have pushed or pulled all the data to or from the hardware. If, however, you configure a non-blocking (IO_NONBLOCKING) device, the functions will process as many bytes as they can and return. They return -IO_WOULDBLOCK if it is not possible to handle any data.

The IO_ASYNC flag makes the system notify the user about the device readiness for reading or writing. These events are received and processed by a user-defined call-back function:

 1void uart_event(IO_io *io, uint16_t event)
 3  if(event & IO_EVENT_READ) {
 4  }
 6  if(event & IO_EVENT_WRITE) {
 7  }
10int main()
12  IO_init();
13  IO_io uart0;
14  IO_uart_init(&uart0, 0, IO_NONBLOCKING|IO_ASYNC, 115200);
15  uart0.event = uart_event;
16  IO_event_enable(&uart0, IO_EVENT_READ|IO_EVENT_WRITE);
17  while(1) IO_wait_for_interrupt();

See test-02-uart-async.c.


The DMA mode allows for transferring data between the peripheral and the main memory in the background. It uses the memory bus when the CPU does not need it for anything else. When in this mode, IO_read and IO_write only initiate a background transfer. The next invocation will either block or return -EWOULDBLOCK, depending on other configuration flags, as long as the current DMA operation is in progress. The memory buffer cannot be changed until the DMA transfer is done. Passing IO_ASYNC will generate completion events for DMA operations. It enables us to implement a pretty neat UART echo app:

 1#include <io/IO.h>
 2#include <io/IO_uart.h>
 4char buffer[30];
 6void uart_event(IO_io *io, uint16_t event)
 8  if(event & IO_EVENT_DMA_READ)
 9    IO_write(io, buffer, 30);
11  if(event & IO_EVENT_DMA_WRITE)
12    IO_read(io, buffer, 30);
15int main()
17  IO_init();
18  IO_io uart0;
19  IO_uart_init(&uart0, 0, IO_DMA|IO_ASYNC, 115200);
20  uart0.event = uart_event;
21  IO_read(&uart0, buffer, 30);
22  while(1) IO_wait_for_interrupt();

See test-03-uart-dma.c.

The driver

There was nothing ultimately hard about writing the driver part. It all boils down to reading the data sheet and following the instruction contained therein. It took quite some time to put everything together into a coherent whole, though. See: TM4C_uart.c.

Table of Contents

  1. Compiling and start-up code
  2. Hardware Abstraction Layer and UART
  3. Debugging, display, heap and fonts
  4. Timers and ADC
  5. DAC, Sound and Nokia Tunes
  6. Random Number Generator, Rendering Engine and the Game
  7. Operating System


I have recently been playing with microcontrollers a lot. Among other things, I have worked through some of the labs from this course on EdX. The material does not use much high-level code, so it gives a good overview of how the software interacts with the hardware. There are some "black box" components in there, though. For me, the best way to learn something well has always been building things from "first principles." I find black boxes frustrating. This post describes the first step on my way to make an Alien Invaders game from "scratch."

Compiling for Tiva

First, we need to be able to compile C code for Tiva. To this end, we will use GCC as a cross-compiler, so make sure you have the arm-none-eabi-gcc command available on your system. We will use the following flags build Tiva-compatible binaries:

  • -mcpu=cortex-m4 - produce the code for ARM Cortex-M4 CPU
  • -mfpu=fpv4-sp-d16 - FPv4 single-precision floating point with the register bank seen by the software as 16 double-words
  • -mfloat-abi=hard - generate floating point instructions and use FPU-specific calling conventions
  • -mthumb - use the Thumb instruction set
  • -std=c11 - use the C11 standard
  • -O0 - don't perform any optimizations
  • -Wall and -pedantic - warn about all the potential issues with the code
  • -ffunction-sections and -fdata-sections - place every function and data item in a separate section in the resulting object file; it allows the optimizations removing all unused code and data to be performed at link-time

Object files

To generate a proper binary image, we need to have some basic understanding of object files produced by the compiler. In short, they consist of sections containing various pieces of compiled code and the corresponding data. These sections may be loadable, meaning that the contents of the section should be read from the object file and stored in memory. They may also be just allocatable, meaning that there is nothing to be loaded, but a chunk of memory needs to be put aside for them nonetheless. There are multiple sections in a typical ELF object file, but we need to know only four of them:

  • .text - contains the program code
  • .rodata - contains the constants (read-only data)
  • .data - contains the read-write data
  • .bss - contains statically allocated variables (initialized to zero)

Let's consider the following code:

 1#include <stdio.h>
 3int a = 12;
 4int b;
 5const char *c = "The quick brown fox jumps over the lazy dog.";
 6const char * const d = "The quick brown fox jumps over the lazy dog.";
 7int e[20];
 8const int f[4] = {7, 4, 2, 1};
10int main(int argc, char **argv)
12  printf("Hello world!\n");
13  return 0;

After compiling it, we end up with an object file containing the following sections (most have been omitted for clarity):

]==> objdump -h test

test:     file format elf64-x86-64

Idx Name          Size      VMA               LMA               File off  Algn
 13 .text         00000192  00000000004003f0  00000000004003f0  000003f0  2**4
 15 .rodata       0000006d  0000000000400590  0000000000400590  00000590  2**4
 24 .data         00000020  0000000000600948  0000000000600948  00000948  2**3
                  CONTENTS, ALLOC, LOAD, DATA
 25 .bss          00000090  0000000000600980  0000000000600980  00000968  2**5

As you can see, every section has two addresses:

  • VMA (virtual memory address) - This is the location of the section the code expects when it runs.
  • LMA (load memory address) - This is the location where the section is stored by the loader.

These two addresses are in most cases the same, except the situation that we care about here: an embedded system. In our binary image, we need put the .data section in ROM because it contains initialized variables whose values would otherwise be lost on reset. The section's LMA, therefore, must point to a location in ROM. However, this data is not constant, so it's final position at program's runtime needs to be in RAM. Therefore, the VMA must point to a location RAM. We will see an example later.

Tiva's memory layout

Tiva has 256K of ROM (range: 0x0000000000-0x0003ffff) and 32K of RAM (range: 0x20000000-0x20003fff). See the table 2-4 on page 90 of the data sheet for details. The NVIC (Interrupt) table needs to be located at address 0x00000000 (section 2.5 of the data sheet). We will create this table in C, put it in a separate object file section, and fill with weak aliases of the default handler function. This approach will enable the user to redefine the interrupt handlers without having to edit the start-up code. The linker will resolve the handler addresses to strong symbols if any are present.

So, we define a dummy interrupt handler that loops indefinitely:

1void __int_handler(void)
3  while(1);

and then create a bunch of weak aliases to this function:

1#define DEFINE_HANDLER(NAME) void NAME ## _handler() __attribute__ ((weak, alias ("__int_handler")))

Finally, we construct the nvic_table, place it in the .nvic section in the resulting object file and fill it with handler addresses:

1#define HANDLER(NAME) NAME ## _handler
2void (*nvic_table[])(void) __attribute__ ((section (".nvic"))) = {
3  HANDLER(reset),
4  HANDLER(nmi),
5  HANDLER(hard_fault),
6  HANDLER(mman),

Linker scripts

We will use linker scripts to set the VMAs and the LMAs to the values we like and to create some symbols whose addresses we can play with in the C code. We first need to define the memory layout:

3  FLASH (rx)  : ORIGIN = 0x00000000, LENGTH = 0x00040000
4  RAM   (rwx) : ORIGIN = 0x20000000, LENGTH = 0x00008000

We then need to tell the linker where to put the section in the final executable:

 3  .text :
 4  {
 5    LONG(0x20007fff)
 6    KEEP(*(.nvic))
 7    *(.text*)
 8    *(.rodata*)
 9     __text_end_vma = .;
10  } > FLASH
12  .data :
13  {
14    __data_start_vma = .;
15    *(.data*)
16    *(vtable)
17    __data_end_vma = .;
18  } > RAM AT > FLASH
20  .bss :
21  {
22    __bss_start_vma = .;
23    *(.bss*)
24    *(COMMON)
25    __bss_end_vma = .;
26  } > RAM
  1. We start with the .text section and begin it with 0x20003fff. It is the initial value of the stack pointer (see the data sheet). Since the stack grows towards lower addresses, we initialize the top of the stack to the last byte of available RAM.
  2. We then put the .nvic section. The KEEP function forces the linker to keep this section even when the link-time optimizations are enabled, and the section seems to be unused. The asterisk in *(.nvic) is a wildcard for an input object file name. Whatever is in the brackets is a wildcard for a section name.
  3. We put all the code and read-only data from all of the input files in this section as well.
  4. We define a new symbol: __text_end_vma and assign its address to the current VMA (the dot means the current VMA).
  5. We put this section in FLASH: > FLASH at line 10.
  6. We combine the .data* sections from all input files into one section and put it behind the .text section in FLASH. We set the VMAs to be in RAM: > RAM AT > FLASH.
  7. Apparently TivaWare changes the value of the VTABLE register and needs to have the NVIC table in RAM, so we oblige: *(vtable).
  8. We put .bss in RAM after .data.
  9. We use asterisks in section names (i.e. .bss*) because -ffunction-sections and -fdata-sections parameters cause the compiler to generate a separate section for each function and data item.

Edit 02.04.2016: The initial stack pointer needs to be aligned to 8 bytes for passing of 64-bit long variadic parameters to work. Therefore, the value of the first four bytes in the text section should be: LONG(0x20007ff8). See this post for details.

See the binutils documentation for more details.

Start-up code

On the system start-up, we need to copy the contents of the .data section from FLASH to RAM ourselves before we can run any code. We do it by defining a reset handler:

 1extern unsigned long __text_end_vma;
 2extern unsigned long __data_start_vma;
 3extern unsigned long __data_end_vma;
 4extern unsigned long __bss_start_vma;
 5extern unsigned long __bss_end_vma;
 7extern void main();
 9void __rst_handler()
11  unsigned long *src = &__text_end_vma;
12  unsigned long *dst = &__data_start_vma;
14  while(dst < &__data_end_vma) *dst++ = *src++;
15  dst = &__bss_start_vma;
16  while(dst < &__bss_end_vma) *dst++ = 0;
18  main();
21void reset_handler() __attribute__ ((weak, alias ("__rst_handler")));

We first declare external symbols. They are put in the symbol table by the linker. The reset handler then moves the .data section from FLASH to RAM, zeroes the .bss section, and calls main.

A test

Let's put everything together. I wrote a short program that blinks an LED using the SysTick interrupt. The color of the LED depends on the switch pressed. The files are here:

Compile and link:

]==> arm-none-eabi-gcc -mcpu=cortex-m4 -mfpu=fpv4-sp-d16 -mfloat-abi=hard -mthumb -std=c11 -O0 -Wall -pedantic -ffunction-sections -fdata-sections -c main.c -g
]==> arm-none-eabi-gcc -mcpu=cortex-m4 -mfpu=fpv4-sp-d16 -mfloat-abi=hard -mthumb -std=c11 -O0 -Wall -pedantic -ffunction-sections -fdata-sections -c TM4C_startup.c -g
]==> arm-none-eabi-ld -T TM4C.ld TM4C_startup.o main.o -o main --gc-sections

Let's see what we have in the resulting binary:

]==> arm-none-eabi-objdump -h main

main:     file format elf32-littlearm

Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000484  00000000  00000000  00010000  2**2
  1 .data         00000004  20000000  00000484  00020000  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000004  20000004  00000488  00020004  2**2

The .text section starts at 0x00000000 both VMA and LMA. The .data section starts at 0x00000484 LMA (in FLASH) but the code expects it to start at 0x20000000 VMA (in RAM). The symbol addresses seem to match the expectations as well:

]==> arm-none-eabi-objdump -t main | grep vma
20000004 g       .bss   00000000 __bss_start_vma
00000484 g       .text  00000000 __text_end_vma
20000008 g       .bss   00000000 __bss_end_vma
20000000 g       .data  00000000 __data_start_vma
20000004 g       .data  00000000 __data_end_vma

We now need to create a raw binary file that we can flash to the board. The arm-none-eabi-objcopy utility can take the relevant sections and put them in an output file aligned according to their LMAs.

]==> arm-none-eabi-objcopy -O binary main main.bin
]==> stat --format=%s main.bin

The total size of the raw binary matches the sum of the sizes of the .text and .data sections (0x488 == 1160). Let's flash it and see if it works!

]==> lm4flash main.bin
Found ICDI device with serial: xxxxxxxx
ICDI version: 9270


Get the full code at GitHub.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion


I have implemented all of the interesting functions listed here and, thus, reached my goal. There were quite a few surprises. I had expected some things to bo more complicated than they are. Conversely, some things that had seemed simple turned out to be quite complex.

  1. I had initially hoped that I would be able to re-use much of glibc and concentrate only on the thread-specific functionality. I was surprised to discover how much of glibc code refers to thread-local storage.
  2. I had expected the interaction between join and detach to be much simpler to handle. Having to implement descriptor caching was an unexpected event.
  3. I had never heard of pthread_once before.
  4. I had not used much of the real-time functionality before, so figuring out the scheduling part was very entertaining. I especially enjoyed implementing the PRIO_INHERIT mutex.

I may revisit this project in the future because there are still some things that I would like to learn more about.

  1. If I'll have the time to learn DWARF, I would like to provide proper .eh_frame for the signal trampoline. It would allow me to implement cancellation using stack unwinding the way glibc does it.
  2. I may look into the inter-process synchronization to learn about the robust futexes.
  3. The Intel article on lock elision seemed interesting, and I'd like to play with this stuff as well.
  4. I may have a look at the compiler-generated TLS.

The End

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion

Condition Variables

Condition variables are a mechanism used for signaling that a certain predicate has become true. The POSIX mechanism for handling them boils down to three functions: cond_wait, cond_signal and cond_broadcast. The first one causes the thread that calls it to wait. The second wakes a thread up so that it can verify whether the condition is true. The third wakes all the waiters up.

The waiter

 4int bseq = cond->broadcast_seq;
 5int futex = cond->futex;
 8while(1) {
 9  st = SYSCALL3(__NR_futex, &cond->futex, FUTEX_WAIT, futex);
10  if(st == -EINTR)
11    continue;
13  tb_futex_lock(&cond->lock);
14  if(cond->signal_num) {
15    --cond->signal_num;
16    goto exit;
17  }
19  if(bseq != cond->broadcast_seq)
20    goto exit;
21  tb_futex_unlock(&cond->lock);

The algorithm is as follows:

  1. We lock the internal lock.
  2. We remember the value of the broadcast sequence and the futex.
  3. In a loop, we wait for the futex.
  4. We go back to sleeping if the FUTEX_WAIT syscall was interrupted by a signal.
  5. We consume a signal, if we can, and exit.
  6. If there was a broadcast, we exit too.
  7. Otherwise, we go back to sleep.


We wake one of the threads up. We bump the value of the futex to prevent a deadlock. Then we bump the number of signals and wake the futex.

 1int tbthread_cond_signal(tbthread_cond_t *cond)
 3  tb_futex_lock(&cond->lock);
 4  if(cond->waiters == cond->signal_num)
 5    goto exit;
 6  ++cond->futex;
 7  ++cond->signal_num;
 8  SYSCALL3(__NR_futex, &cond->futex, FUTEX_WAKE, 1);
10  tb_futex_unlock(&cond->lock);
11  return 0;


We wake all the waiters. The algorithm is essentially the same as for signal, except that, we bump the broadcast sequence number and wake all the threads instead of just one.

 1int tbthread_cond_broadcast(tbthread_cond_t *cond)
 3  tb_futex_lock(&cond->lock);
 4  if(!cond->waiters)
 5    goto exit;
 6  ++cond->futex;
 7  ++cond->broadcast_seq;
 8  SYSCALL3(__NR_futex, &cond->futex, FUTEX_WAKE, INT_MAX);
10  tb_futex_unlock(&cond->lock);
11  return 0;

See the full patch at GitHub.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion

RW Locks

A read-write lock protects a critical section by allowing multiple readers when there are no writers. We won't bother implementing lock attributes handling because we don't support process-shared locks (irrelevant in our case) and we don't let the user prefer readers (a non-POSIX extension). The implementation remembers the ID of the current writer if any. It also counts readers as well as the queued writers. We use two futexes, one to block the readers and one to block the writers.

A writer first bumps the number of queued writers. If there is no other writer and no readers, it marks itself as the owner of the lock and decrements the number of queued writers. It goes to sleep on the writer futex otherwise.

 1int tbthread_rwlock_wrlock(tbthread_rwlock_t *rwlock)
 3  int queued = 0;
 4  while(1) {
 5    tb_futex_lock(&rwlock->lock);
 7    if(!queued) {
 8      queued = 1;
 9      ++rwlock->writers_queued;
10    }
12    if(!rwlock->writer && !rwlock->readers) {
13      rwlock->writer = tbthread_self();
14      --rwlock->writers_queued;
15      tb_futex_unlock(&rwlock->lock);
16      return 0;
17    }
18    int sleep_status = rwlock->wr_futex;
20    tb_futex_unlock(&rwlock->lock);
22    SYSCALL3(__NR_futex, &rwlock->wr_futex, FUTEX_WAIT, sleep_status);
23  }

A reader acquires the lock if there are no writers at all. It goes to sleep on the reader futex otherwise.

 1int tbthread_rwlock_rdlock(tbthread_rwlock_t *rwlock)
 3  while(1) {
 4    tb_futex_lock(&rwlock->lock);
 6    if(!rwlock->writer && !rwlock->writers_queued) {
 7      ++rwlock->readers;
 8      tb_futex_unlock(&rwlock->lock);
 9      return 0;
10    }
11    int sleep_status = rwlock->rd_futex;
13    tb_futex_unlock(&rwlock->lock);
15    SYSCALL3(__NR_futex, &rwlock->rd_futex, FUTEX_WAIT, sleep_status);
16  }

When unlocking, we use the writer field to determine whether we were a reader or a writer. If we were a writer, we've had an exclusive ownership of the lock. Therefore, we need to either wake another writer or all of the readers, depending on the state of the counters. If we were a reader, we've had a non-exclusive lock. Therefore, we only need to wake a writer when we're the last reader and there is a writer queued. We bump the value of the futex because we want to handle the cases when FUTEX_WAKE was called before the other thread manged to call FUTEX_WAIT.

 1int tbthread_rwlock_unlock(tbthread_rwlock_t *rwlock)
 3  tb_futex_lock(&rwlock->lock);
 4  if(rwlock->writer) {
 5    rwlock->writer = 0;
 6    if(rwlock->writers_queued) {
 7      __sync_fetch_and_add(&rwlock->wr_futex, 1);
 8      SYSCALL3(__NR_futex, &rwlock->wr_futex, FUTEX_WAKE, 1);
 9    } else {
10      __sync_fetch_and_add(&rwlock->rd_futex, 1);
11      SYSCALL3(__NR_futex, &rwlock->rd_futex, FUTEX_WAKE, INT_MAX);
12    }
13    goto exit;
14  }
16  --rwlock->readers;
17  if(!rwlock->readers && rwlock->writers_queued) {
18    __sync_fetch_and_add(&rwlock->wr_futex, 1);
19    SYSCALL3(__NR_futex, &rwlock->wr_futex, FUTEX_WAKE, 1);
20  }
23  tb_futex_unlock(&rwlock->lock);
24  return 0;

See the full patch at GitHub.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion

Thread Scheduling

The scheduler makes a decision of which thread to run next based on two parameters: scheduling policy and priority.

Conceptually, the scheduler maintains a list of runnable threads for each possible sched_priority value. In order to determine which thread runs next, the scheduler looks for the nonempty list with the highest static priority and selects the thread at the head of this list.

A thread's scheduling policy determines where it will be inserted into the list of threads with equal static priority and how it will move inside this list. the sched(7) man page

The supported policies are:

  • SCHED_NORMAL - It's the default Linux policy for threads not requiring any real-time machinery. All the threads have priority of 0, and the decision of which thread gets run next is based on the nice mechanism.
  • SCHED_FIFO - The threads have priority from 1 (low) to 99 (high). When a SCHED_FIFO thread becomes runnable, it will preempt any thread of lower priority. There is no time slicing, so the hight priority thread runs as long as it must.
  • SCHED_RR - RR stands for round-robin. It's the same as SCHED_FIFO except each thread is allowed to run for a limited time quantum.

See the sched(7) man page for more details.

Setting the scheduling parameters of a kernel task boils down to invoking the sys_sched_setscheduler syscall, as shown below.

 1struct tb_sched_param
 3  int sched_priority;
 6int tb_set_sched(tbthread_t thread, int policy, int priority)
 8  struct tb_sched_param p; p.sched_priority = priority;
 9  int ret = SYSCALL3(__NR_sched_setscheduler, thread->tid, policy, &p);
10  if(!ret)
11    thread->sched_info = SCHED_INFO_PACK(policy, priority);
12  return ret;

There is a bit more fun to it, though. As you can see, the kernel can only schedule a task that already exists. Therefore, we need to have a way to set the thread's priority before this thread invokes the user function. The reason for this is that we may need to abort this thread immediately should the scheduler setting fail. We do it by having another futex that we wake when we know whether the thread can run or not:

1if(th->start_status != TB_START_OK) {
2  SYSCALL3(__NR_futex, &th->start_status, FUTEX_WAIT, TB_START_WAIT);
3  if(th->start_status == TB_START_EXIT)
4    SYSCALL1(__NR_exit, 0);

This futex is initialized in the tbthread_create function depending on the thread attributes:

2  (*thread)->start_status = TB_START_WAIT;
3else {
4  tbthread_t self = tbthread_self();
5  (*thread)->sched_info = self->sched_info;

And then set to either TB_START_OK or TB_START_EXIT after we spawn the thread but before tbthread_create exits:

 1if(!attr->sched_inherit) {
 2  ret = tb_set_sched(*thread, attr->sched_policy, attr->sched_priority);
 4  if(ret) (*thread)->start_status = TB_START_EXIT;
 5  else (*thread)->start_status = TB_START_OK;
 6  SYSCALL3(__NR_futex, &(*thread)->start_status, FUTEX_WAKE, 1);
 8  if(ret) {
 9    wait_for_thread(*thread);
10    goto error;
11  }

See the patch at GitHub.

Priority mutexes

Priority mutexes come in three varieties:

  • TBTHREAD_PRIO_NONE - Acquiring this type of mutex does not change the scheduling characteristics of the thread.
  • TBTHREAD_PRIO_INHERIT - When the mutex owner blocks another thread with a higher priority, the owner inherits the priority of the thread it blocks if it's higher than its own.
  • TBTHREAD_PRIO_PROTECT - Acquiring this kind of mutex raises the priority of the owner to the prioceiling value of the mutex.

Thread Bites implements this functionality by keeping lists of PRIO_INHERIT and PRIO_PROTECT mutexes. It then calculates the highest possible priority taking into account the priority of the mutexes and the priority set by the user.

The implementation of PRIO_PROTECT is relatively straightforward. Whenever a thread acquires this kind of mutex, it is added to the list, and the priority of the thread is recalculated:

 1static int lock_prio_protect(tbthread_mutex_t *mutex)
 3  lock_prio_none(mutex);
 4  tb_protect_mutex_sched(mutex);
 5  return 0;
 8static int unlock_prio_protect(tbthread_mutex_t *mutex)
10  tbthread_t self = tbthread_self();
11  tb_protect_mutex_unsched(mutex);
12  unlock_prio_none(mutex);
13  return 0;

Implementing PRIO_INHERIT is a lot more tricky. We add the mutex to the appropriate list when a thread acquires it. Whenever a higher priority thread tries to lock the mutex, it bumps the priority of the blocker. But the priority recalculation is done only at this point. Implementing it like this covers all the main cases and is not horrendously hard. It allows for simple recursion: if the owner of a mutex gets blocked, the blocker inherits the priority that comes with the first mutex. It also has a couple of drawbacks:

  • It assumes that the kernel will always wake the highest priority thread. It makes sense and is most likely the case. However, I have not tested it.
  • If the owner of a PRIO_INHERIT mutex is already blocked on another mutex of the same kind and it's priority gets bumped later, the last thread in the line won't be affected.
 1static int lock_prio_inherit(tbthread_mutex_t *mutex)
 3  tbthread_t self = tbthread_self();
 5  while(1) {
 6    int locked = 0;
 7    tb_futex_lock(&mutex->internal_futex);
 8    if(mutex->futex == 0) {
 9      locked = 1;
10      mutex->owner = self;
11      mutex->futex = 1;
12      tb_inherit_mutex_add(mutex);
13    }
14    else
15      tb_inherit_mutex_sched(mutex, self);
16    tb_futex_unlock(&mutex->internal_futex);
17    if(locked)
18      return 0;
19    SYSCALL3(__NR_futex, &mutex->futex, FUTEX_WAIT, 1);
20  }
23static int unlock_prio_inherit(tbthread_mutex_t *mutex)
25  tb_futex_lock(&mutex->internal_futex);
26  tb_inherit_mutex_unsched(mutex);
27  mutex->owner = 0;
28  mutex->futex = 0;
29  SYSCALL3(__NR_futex, &mutex->futex, FUTEX_WAKE, 1);
30  tb_futex_unlock(&mutex->internal_futex);
31  return 0;

It was by far the most challenging part so far. See the patch at GitHub.

Remaining functions

  • pthread_setconcurrency- It defines how many kernel tasks should be created to handle the user-level threads. It does not make sense in our case because we create a kernel task for every thread.
  • pthread_attr_setscope - It defines the set of threads against which the thread will compete for resources. There are two settings: PTHREAD_SCOPE_SYSTEM meaning all the threads in the entire system and PTHREAD_SCOPE_PROCESS meaning only the threads within the process. The man page says that Linux only supports PTHREAD_SCOPE_SYSTEM, but I am not sure whether it's still the case with all the cgroups stuff.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion


Cancellation boils down to making one thread exit following a request from another thread. It seems that calling tbthread_exit at an appropriate point is enough to implement all of the behavior described in the man pages. We will go this way despite the fact that it is not the approach taken by glibc. Glibc unwinds the stack back to the point invoking the user-supplied thread function. This behavior allows it to simulate an exception if C++ code is using the library. We don't bother with C++ support for the moment and don't always care to supply valid DWARF information. Therefore, we will take the easier approach.

tbthread_setcancelstate and tbthread_setcanceltype are the two functions controlling the response of a thread to a cancellation request. The former enables or disables cancellation altogether queuing the requests for later handling if necessary. The latter decides whether the thread should abort immediately or at a cancellation point. POSIX has a list of cancellation points, but we will not bother with them. Instead, we'll just use tbthread_testcancel and the two functions mentioned before for this purpose.

The thread must not get interrupted after it disables or defers cancellation. It would likely lead to deadlocks due to unreleased mutexes, memory leaks and such. The trick here is to update all the cancellation related flags atomically. So, we use one variable to handle the following flags:

  • TB_CANCEL_ENABLED: The cancellation is enabled; if a cancellation request has been queued, reaching a cancellation point will cause the thread to exit.
  • TB_CANCEL_DEFERRED: The cancellation is deferred (not asynchronous); SIGCANCEL will not be sent; see the paragraph on signal handling.
  • TB_CANCELING: A cancellation request has been queued; depending on other flags, SIGCANCEL may be sent.
  • TB_CANCELED: A cancellation request has been taken into account and the thread is in the process of exiting; this flag is used to handle the cases when a cancellation point has been reached before SIGCANCEL has been delivered by the kernel.

The tbhread_testcancel looks as follows:

 1void tbthread_testcancel()
 3  tbthread_t thread = tbthread_self();
 4  uint8_t val, newval;
 6  while(1) {
 7    newval = val = thread->cancel_status;
 8    if(!(val & TB_CANCEL_ENABLED) || !(val & TB_CANCELING) ||
 9       (val & TB_CANCELED))
10      return;
11    newval |= TB_CANCELED;
12    if(__sync_bool_compare_and_swap(&thread->cancel_status, val, newval))
13      break;
14  }
15  tbthread_exit(TBTHREAD_CANCELED);

See the full patch at GitHub.

Clean-up handlers

The user may register a bunch of functions cleaning up the mess caused by an unexpected interruption. They are installed with tbthread_cleanup_push and called when the thread exits abnormally. The purpose of these functions is to unlock mutexes, free the heap memory and such. tbthread_cleanup_pop removes them and optionally executes in the process.

 1void tbthread_cleanup_push(void (*func)(void *), void *arg)
 3  tbthread_t self = tbthread_self();
 4  struct cleanup_elem *e = malloc(sizeof(struct cleanup_elem));
 5  e->func = func;
 6  e->arg = arg;
 7  list_add_elem(&self->cleanup_handlers, e, 1);
10void tbthread_cleanup_pop(int execute)
12  tbthread_t self = tbthread_self();
13  list_t *node = self->;
14  if(!node)
15    return;
16  list_rm(node);
17  struct cleanup_elem *e = (struct cleanup_elem*)node->element;
18  if(execute)
19    (*e->func)(e->arg);
20  free(e);
21  free(node);

See the full patch at GitHub.

Signals and asynchronous cancellation

The asynchronous cancellation uses the first real-time signal, SIGRTMIN, that we call SIGCANCEL here for clarity.

Registering a signal handler is somewhat more tricky than just calling the appropriate syscall. It is so because, on x86_64, we need to provide a function that restores the stack after the signal handler returns. The function is called a signal trampoline and its purpose is to invoke sys_rt_sigreturn. The trampoline is registered with the kernel using a special sigaction flag:

1void __restore_rt();
2#define SA_RESTORER 0x04000000
4int tbsigaction(int signum, struct sigaction *act, struct sigaction *old)
6  act->sa_flags |= SA_RESTORER;
7  act->sa_restorer = __restore_rt;
8  return SYSCALL4(__NR_rt_sigaction, signum, act, old, sizeof(sigset_t));

The trampoline itself, called __restore_rt here, is defined in assembly as follows:

1  .text
3  .global __restore_rt
4  .type   __restore_rt,@function
5  .align  16
8  movq $__NR_rt_sigreturn, %rax
9  syscall

Looking at the corresponding glibc code, you can see that they add the eh_frame info here. The comments say that it is to aid gdb and handle the stack unwinding. I don't know enough DWARF to write one on my own, gdb does not seem to be utterly confused without it, and we won't do stack unwinding, so we just won't bother with it for the moment.

In the cancellation handler, we first check whether it's the right signal and that it has been sent by a thread in the same thread group. We then need to check whether the thread is still in the asynchronous cancellation mode. It might have changed between the time the signal was sent and the time the it is delivered. Finally, we call thread_testcancel to see if the thread should exit.

 1void tb_cancel_handler(int sig, siginfo_t *si, void *ctx)
 3  if(sig != SIGCANCEL || si->si_pid != tb_pid || si->si_code != SI_TKILL)
 4    return;
 6  tbthread_t self = tbthread_self();
 7  if(self->cancel_status & TB_CANCEL_DEFERRED)
 8    return;
10  tbthread_testcancel();

We invoke sys_tgkill to send the signal:

1SYSCALL3(__NR_tgkill, tb_pid, thread->exit_futex, SIGCANCEL);

See the full patch at GitHub.

Cancellation of a "once" function

The implementation of tbthread_once gets quite a bit more interesting as well. If the thread invoking the initialization function gets canceled, another thread needs to pick it up. We need to install a cleanup handler that will change the state of the once control back to TB_ONCE_NEW and wake all the threads so that they could restart from the beginning:

 1static void once_cleanup(void *arg)
 3  tbthread_once_t *once = (tbthread_once_t *)arg;
 4  *once = TB_ONCE_NEW;
 5  SYSCALL3(__NR_futex, once, FUTEX_WAKE, INT_MAX);
 8int tbthread_once(tbthread_once_t *once, void (*func)(void))
10  if(!once || !func)
11    return -EINVAL;
13  int cancel_state;
15  while(1) {
16    if(*once == TB_ONCE_DONE)
17      return 0;
19    //--------------------------------------------------------------------------
20    // The executor
21    //--------------------------------------------------------------------------
22    tbthread_setcancelstate(TBTHREAD_CANCEL_DISABLE, &cancel_state);
23    if(__sync_bool_compare_and_swap(once, TB_ONCE_NEW, TB_ONCE_IN_PROGRESS)) {
24      tbthread_cleanup_push(once_cleanup, once);
25      tbthread_setcancelstate(cancel_state, 0);
27      (*func)();
29      tbthread_setcancelstate(TBTHREAD_CANCEL_DISABLE, &cancel_state);
30      tbthread_cleanup_pop(0);
32      *once = TB_ONCE_DONE;
33      SYSCALL3(__NR_futex, once, FUTEX_WAKE, INT_MAX);
34      tbthread_setcancelstate(cancel_state, 0);
35      return 0;
36    }
38    tbthread_setcancelstate(cancel_state, 0);
40    //--------------------------------------------------------------------------
41    // The waiters
42    //--------------------------------------------------------------------------
43    while(1) {
44      SYSCALL3(__NR_futex, once, FUTEX_WAIT, TB_ONCE_IN_PROGRESS);
45      if(*once != TB_ONCE_IN_PROGRESS)
46        break;
47    }
48  }

See the patch at GitHub.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion

Recycling the thread descriptors

How do we know when the thread task has died? This is what the CHILD_SETTID and CHILD_CLEARTID flags to sys_clone are for. If they are set, the kernel will store the new thread's TID at the location pointed to by the ctid argument (see tb #1). When the thread terminates, the kernel will set the TID to 0 and wake the futex at this location. It is a convenient way to wait for a thread to finish. Unfortunately, as far as I can tell, there is no way to unset these flags, and it makes implementing tbthread_detach a pain. We cannot delete the thread descriptor in the thread it refers to anymore. Doing so would cause the kernel to write to a memory location that might have been either unmapped or reused. Therefore, we need to have some sort of a cache holding thread descriptors and make sure that we re-use them only after the thread they were referring to before has exited. Thread bites uses two linked lists to maintain this cache, and the descriptor allocation function calls the following procedure to wait until the corresponding task is gone:

1static void wait_for_thread(tbthread_t thread)
3  uint32_t tid = thread->exit_futex;
4  long ret = 0;
5  if(tid != 0)
6    do {
7      ret = SYSCALL3(__NR_futex, &thread->exit_futex, FUTEX_WAIT, tid);
8    } while(ret != -EWOULDBLOCK && ret != 0);

See the full patch at GitHub.

Joining threads

Joining complicates things a bit further because it does quite a bit of error checking to prevent deadlocks and such. To perform these checks, the thread calling tbthread_join needs to have a valid thread descriptor obtainable by tbthread_self. The problem is that we have never set this thread descriptor up for the main thread, and we need to do it by hand at the beginning of the program. The original state needs to be restored at the end because glibc uses it internally and not cleaning things up causes segfaults.

 1static void *glibc_thread_desc;
 2void tbthread_init()
 4  glibc_thread_desc = tbthread_self();
 5  tbthread_t thread = malloc(sizeof(struct tbthread));
 6  memset(thread, 0, sizeof(struct tbthread));
 7  thread->self = thread;
 8  SYSCALL2(__NR_arch_prctl, ARCH_SET_FS, thread);
11void tbthread_finit()
13  free(tbthread_self());
14  SYSCALL2(__NR_arch_prctl, ARCH_SET_FS, glibc_thread_desc);

After performing all the validity and deadlock checks, the meat of tbthread_join is rather simple:

3  *retval = thread->retval;
5return 0;

See the full patch at GitHub.

Dynamic initialization

pthread_once is an interesting beast. Its purpose is to initialize dynamically some resources by calling a designated function exactly once. The fun part is that the actual initialization call may be made from multiple threads at the same time. pthread_once_t, therefore, is kind of like a mutex, but has three states instead of two:

  • new: the initialization function has not been called yet; one of the threads needs to call it.
  • in progress: the initialization function is running; the threads are waiting for it to finish.
  • done: the initialization function is done; all the threads may be woken up.

The thread that manages to change the state from new to in progress gets to call the function. All the other threads wait until the done state is reached.

 1int tbthread_once(tbthread_once_t *once, void (*func)(void))
 3  if(!once || !func)
 4    return -EINVAL;
 6  if(*once == TB_ONCE_DONE)
 7    return 0;
 9  if(__sync_bool_compare_and_swap(once, TB_ONCE_NEW, TB_ONCE_IN_PROGRESS)) {
10    (*func)();
11    *once = TB_ONCE_DONE;
12    SYSCALL3(__NR_futex, once, FUTEX_WAKE, INT_MAX);
13    return 0;
14  }
16  while(*once != TB_ONCE_DONE)
18  return 0;

Side effects

The original glibc thread descriptor stores the localization information for the thread. Changing it to ours causes seemingly simple functions, like strerror, to segfault. Therefore, we need to implement strerror ourselves.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion


This part discusses an implementation of a mutex. It will not be a particularly efficient mutex, but it will be an understandable and straightforward one. We will not bother minimizing the number of system calls or implementing lock elision. We will also not handle the case of inter-process communication. Therefore, the process-shared and robust mutexes will not be discussed. If you are interested in these ideas, I recommend the kernel's documentation file on robust mutexes and Intel's blog post on lock elision. The scheduling related parameters will likely be dealt with on another occasion.


POSIX defines a bunch of different mutexes. See the manpage for pthread_mutexattr_settype to learn more. On Linux, all of them are implemented using the same locking primitive - a Futex (Fast User-Space Mutex). It is a 4-byte long chunk of aligned memory. Its contents can be updated atomically, and its address can be used to refer to a kernel-level process queue. The kernel interface is defined as follows:

1asmlinkage long sys_futex(u32 __user *uaddr, int op, u32 val,
2                         struct timespec __user *utime, u32 __user *uaddr2,
3                         u32 val3);

We will only use the first four out of the six parameters here. The first one is the address of the futex, and the second one is the type of the operation to be performed. The meaning of the remaining parameters depends on the context. We will need only two of the available operations to implement a mutex:

  • FUTEX_WAIT puts a thread to sleep if the value passed in val is the same as the value stored in the memory pointed to by *uaddr. Optionally, the sleep time may be limited by passing a pointer to a timespec object. The return values are:
    • 0, if the thread was woken up by FUTEX_WAIT.
    • EWOULDBLOCK, if the value of the futex was different than val.
    • EINTR, if the sleep was interrupted by a signal.
  • FUTEX_WAKE wakes the number of threads specified in val. In practice, it only makes sense to either wake one or all sleeping threads, so we pass either 1 or INT_MAX respectively.

See the original futex paper for more details.

Normal mutex

We start with a normal mutex because it is possible to implement all the other kinds using the procedures defined for it. If the value of the associated futex is 0, then the mutex is unlocked. Locking it means changing the value to 1. To avoid race conditions, both the checking and the changing need to be done atomically. GCC has a built-in function to do this that results with lock cmpxchgq or similar being emitted in assembly. If the locking fails, we need to wait until another thread releases the mutex and re-check the lock.

1static int lock_normal(tbthread_mutex_t *mutex)
3  while(1) {
4    if(__sync_bool_compare_and_swap(&mutex->futex, 0, 1))
5      return 0;
6    SYSCALL3(__NR_futex, &mutex->futex, FUTEX_WAIT, 1);
7  }

The logic of trylock is essentially the same, except that we return a failure instead of sleeping.

1static int trylock_normal(tbthread_mutex_t *mutex)
3  if(__sync_bool_compare_and_swap(&mutex->futex, 0, 1))
4      return 0;
5  return -EBUSY;

To unlock a mutex, we do the reverse. We change the futex value from 1 to 0 and wake one thread waiting for the futex to be released.

1static int unlock_normal(tbthread_mutex_t *mutex)
3  if(__sync_bool_compare_and_swap(&mutex->futex, 1, 0))
4    SYSCALL3(__NR_futex, &mutex->futex, FUTEX_WAKE, 1);
5  return 0;

Note that the values stored in the futex are application-specific and arbitrary. The kernel does not care and does not change this variable except in one case, which we will discuss in a later chapter.

This mutex is not particularly efficient because we make a system call while unlocking regardless of whether there is a waiting thread or not. To see how the situation could be improved, please refer to Ulrich Drepper's Futexes Are Tricky.

Error-check mutex

These guys are the same as the ones discussed earlier, except that they do additional bookkeeping. An error-check mutex remembers who owns it, to report the following types of errors:

  • re-locking of a mutex the thread already owns
  • unlocking a mutex owned by another thread
  • unlocking a mutex that is not locked

The behavior of normal mutexes in these cases is undefined.

 1static int lock_errorcheck(tbthread_mutex_t *mutex)
 3  tbthread_t self = tbthread_self();
 4  if(mutex->owner == self)
 5    return -EDEADLK;
 6  lock_normal(mutex);
 7  mutex->owner = self;
 8  return 0;
11static int trylock_errorcheck(tbthread_mutex_t *mutex)
13  int ret = trylock_normal(mutex);
14  if(ret == 0)
15    mutex->owner = tbthread_self();
16  return ret;
19static int unlock_errorcheck(tbthread_mutex_t *mutex)
21  if(mutex->owner != tbthread_self() || mutex->futex == 0)
22    return -EPERM;
23  mutex->owner = 0;
24  unlock_normal(mutex);
25  return 0;

Recursive mutex

Recursive mutexes may be locked multiple times by the same thread and require the same numbers of unlock operations to be released. To provide this kind of functionality, we just need to add a counter counter.

 1static int lock_recursive(tbthread_mutex_t *mutex)
 3  tbthread_t self = tbthread_self();
 4  if(mutex->owner != self) {
 5    lock_normal(mutex);
 6    mutex->owner   = self;
 7  }
 8  if(mutex->counter == (uint64_t)-1)
 9    return -EAGAIN;
10  ++mutex->counter;
11  return 0;
14static int trylock_recursive(tbthread_mutex_t *mutex)
16  tbthread_t self = tbthread_self();
17  if(mutex->owner != self && trylock_normal(mutex))
18    return -EBUSY;
20  if(mutex->owner != self) {
21    mutex->owner = self;
22    mutex->counter = 1;
23    return 0;
24  }
26  if(mutex->counter == (uint64_t)-1)
27    return -EAGAIN;
29  ++mutex->counter;
30  return 0;
33static int unlock_recursive(tbthread_mutex_t *mutex)
35  if(mutex->owner != tbthread_self())
36    return -EPERM;
37  --mutex->counter;
38  if(mutex->counter == 0) {
39    mutex->owner = 0;
40    return unlock_normal(mutex);
41  }
42  return 0;

Other code

This is pretty much it. The remaining part of the code is not interesting. Both mutex and mutexattr objects need to be initialized to their default values, but the futexes don't need any initialization or cleanup. As always, the full patch is available on GitHub.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion


The second part of our threading journey covers the thread-local storage. I do not mean here the compiler generated TLS that I mentioned in the first part. I mean the stuff that involves pthread_setspecific and friends. I don't think it's the most critical part of all this threading business. I barely ever use it in practice. However, we will need to refer to the current thread all over the place, and this requires the TLS. It's better to deal with it once and for all, especially that it's not particularly complicated.


How does one store something in a fixed location and make it distinct for each execution thread? The only two answers that come to my mind are either using syscalls to associate something with kernel's task_struct or using CPU registers. The first approach requires context switches to retrieve the value, so it's rather inefficient. The second option, though, should be pretty fast. Conveniently, on x86_64, some registers are left unused (see StackOverflow). In fact, the SETTLS option to clone takes a pointer and puts it in the file segment register for us, so we don't even need to make an extra syscall just for that.

Since fs is a segment register, we cannot retrieve it's absolute value without engaging the operating system. Linux on x86_64 uses the arch_prctl syscall for this purpose:

1#include "tb.h"
2#include <asm/prctl.h>
4tbthread_t tbthread_self()
6  tbthread_t self;
7  SYSCALL2(__NR_arch_prctl, ARCH_GET_FS, &self);
8  return self;

This syscall seems expensive (link) and making it defies the reason for using a register in the first place. We can read the memory at an address relative to the value of the register, though. Using this fact, we can make our thread struct point to itself and then retrieve this pointer using inline assembly. Here's how:

1tbthread_t tbthread_self()
3  tbthread_t self;
4  asm("movq %%fs:0, %0\n\t" : "=r" (self));
5  return self;

For the main thread, the linker initializes the TLS segment for pthreads automatically using arch_prctl. We cannot use it for our purposes, but we can count on tbthread_self returning a unique, meaningful value.

See the full patch here.


The actual TLS is handled by the following four functions: pthread_key_create, pthread_key_delete, pthread_getspecific, pthread_setspecific. I will not explain what they do because it should pretty self-evident. If it's not, see the man pages.

In principle, we could just have a hash table in the tbthread struct. We could then use setspecific and getspecific to set and retrieve the value associated with each given key. Calling setspecific with a NULL pointer would delete the key. Unfortunately, the designers of pthreads made it a bit more complicated by having separate key_create and key_delete functions, with the delete function invalidating the key in all the threads. Glibc uses a global array of keys and sequence numbers in a clever way to solve this problem. We will take almost the same approach in a less efficient but a bit clearer way.

We will have a global array representing keys. Each element of this array will host a pointer to a destructor and a sequence number. The index in the array will be the actual key passed around between the TLS functions. Both key_create and key_delete will bump the sequence number associated with a key in an atomic way. If the number is even, the key is not allocated, if the number is odd, it is.

 1static struct
 3  uint64_t seq;
 4  void (*destructor)(void *);
 7#define KEY_UNUSED(k) ((keys[k].seq&1) == 0)
 8#define KEY_ACQUIRE(k) (__sync_bool_compare_and_swap(&(keys[k].seq), keys[k].seq, keys[k].seq+1))
 9#define KEY_RELEASE(k) (__sync_bool_compare_and_swap(&(keys[k].seq), keys[k].seq, keys[k].seq+1))
11int tbthread_key_create(tbthread_key_t *key, void (*destructor)(void *))
13  for(int i = 0; i < TBTHREAD_MAX_KEYS; ++i) {
14    if(KEY_UNUSED(i) && KEY_ACQUIRE(i)) {
15      *key = i;
16      keys[i].destructor = destructor;
17      return 0;
18    }
19  }
20  return -ENOMEM;

Each tbthread struct will hold an array of the same size as the global array of keys. Each element in this array will hold a data pointer and a sequence number. Storing data will set the sequence number as well. Retrieving the data will check whether the local and the global sequence number match before proceeding.

 1void *tbthread_getspecific(tbthread_key_t key)
 3  if(key >= TBTHREAD_MAX_KEYS || KEY_UNUSED(key))
 4    return 0;
 6  tbthread_t self = tbthread_self();
 7  if(self->tls[key].seq == keys[key].seq)
 8    return self->tls[key].data;
 9  return 0;

See the full patch here.


KEY_ACQUIRE and KEY_RELEASE use gcc atomic builtins described here.

Table of Contents

  1. Creating threads
  2. Thread-local storage
  3. Mutexes
  4. Joining and initialization
  5. Cancellation
  6. Scheduling
  7. RW Locks
  8. Condition Variables
  9. Conclusion


This is a first of hopefully many posts documenting my attempts to understand how to implement a pthread-style threading system on Linux. To this end, I started implementing a small, non-portable and a rather useless library that I called Thread Bites. Thread Bites is useless mainly because it lacks support for compiler-generated thread-local storage. It may not sound like a grave issue, but it makes Thread Bites incompatible with most of glibc. Therefore, I had to provide my own functionality for invoking syscalls, managing the heap and even printing stuff to stdout. It's all pretty simple and understandable so far, so I hope I will be able to implement most of the pthreads' functionality in a couple of small bites. You can get the source from GitHub.


For a program to be even remotely useful, it needs to communicate with the user in one way or another. A standard library for the programming language, like glibc, typically provides all the necessary components for such communication. For reasons mentioned in the introduction, using glibc in this case is not advisable. Hence, I need to find a way to call the operating system directly without using the syscall function, because it is also a part glibc and sets errno, which is supposed to reside in the TLS that I did not set up.

All that is needed to implement an equivalent function is shuffling around the values of the registers to translate between the calling conventions of C and the Linux kernel, as described here and here on page 20. As it turns out, it's not that hard to do it using inline assembly in C, and there's an excellent tutorial here.

 1#define SYSCALL(name, a1, a2, a3, a4, a5, a6)           \
 2  ({                                                    \
 3    long result;                                        \
 4    long __a1 = (long)(a1);                             \
 5    long __a2 = (long)(a2);                             \
 6    long __a3 = (long)(a3);                             \
 7    long __a4 = (long)(a4);                             \
 8    long __a5 = (long)(a5);                             \
 9    long __a6 = (long)(a6);                             \
10    register long _a1 asm("rdi") = __a1;                \
11    register long _a2 asm("rsi") = __a2;                \
12    register long _a3 asm("rdx") = __a3;                \
13    register long _a4 asm("r10") = __a4;                \
14    register long _a5 asm("r8")  = __a5;                \
15    register long _a6 asm("r9")  = __a6;                \
16    asm volatile (                                      \
17      "syscall\n\t"                                     \
18      : "=a" (result)                                   \
19      : "0" (name), "r" (_a1), "r" (_a2), "r" (_a3),    \
20        "r" (_a4), "r" (_a5), "r" (_a6)                 \
21      : "memory", "cc", "r11", "cx");                   \
22    (long) result; })
24#define SYSCALL1(name, a1) \
25  SYSCALL(name, a1, 0, 0, 0, 0, 0)
26#define SYSCALL2(name, a1, a2) \
27  SYSCALL(name, a1, a2, 0, 0, 0, 0)
28#define SYSCALL3(name, a1, a2, a3) \
29  SYSCALL(name, a1, a2, a3, 0, 0, 0)
30#define SYSCALL4(name, a1, a2, a3, a4) \
31  SYSCALL(name, a1, a2, a3, a4, 0, 0)
32#define SYSCALL5(name, a1, a2, a3, a4, a5) \
33  SYSCALL(name, a1, a2, a3, a4, a5, 0)
34#define SYSCALL6(name, a1, a2, a3, a4, a5, a6) \
35  SYSCALL(name, a1, a2, a3, a4, a5, a6)

All this is, of course, horribly inefficient because it messes up with all the registers even in the situations it does not have to. The intermediate variables for the parameters (__a1 and friends) are used to prevent embedded function calls from messing with the registers that have already been set; think of strlen in SYSCALL3(__NR_write, 1, blah, strlen(blah)).

See the full code on GitHub.

Printing to stdout

It seems that glibc is using errno and other thread local stuff to calculate buffer sizes in one of the subroutines called by printf. It causes printf to segfault when called concurrently from different threads because of the same TLS story. Thread Bites provides a convenience function similar to printf and supporting %s %x %u %o %d in l and ll flavors:

1void tbprint(const char *format, ...);


Glibc's default malloc implementation, a ptmalloc2 derivative, uses thread-specific arenas to limit lock congestion caused by calling malloc concurrently from multiple threads. It looks like it depends on TLS, so using it is not the best idea. Thread Bites comes with its own evil version of malloc. It's pathological because it's extremely prone to fragmentation, it never shrinks the heap, and it's essentially one big critical section. It has some undeniable advantages too: it works, it's incredibly simple, and it fits in around 50 lines of code. Look here to find it.

The only thing worth noting in this section is that the sys_brk syscall does not behave like glibc's brk or sbrk functions. On error, it would return the previous location of the heap boundary, so the code calls it with an obviously wrong parameter (0) to figure out what the initial heap boundary is.


Clone is an interesting beast. From the standpoint of this section, the relevant thing about it is that it behaves mostly like fork, except that the child is launched on a new stack. For this reason and to generate proper Call Frame Information (see here and here), it needs to be implemented in assembly. The implementation puts the user function pointer and its argument on the child's stack so that they can be later popped and called by the child. Then, it puts the syscall parameters in the appropriate registers and makes the syscall. Again, all the code is on GitHub.

Creating a thread

After dealing with all this boilerplate, the actual thread creation is relatively straightforward. First, the new thread needs a stack it can run on. The stack could be allocated on the heap, but it's probably safer to get a separate memory region for it. It can be done using the mmap syscall. A proper threading library should check the system limits to figure out the size of the stack, but Thread Bites is not proper, so it will use 8MiB as a default. All the function parameters in the snippet below match glibc's mmap call.

1void *stack = tbmmap(NULL, attr->stack_size, PROT_READ | PROT_WRITE,
2                     MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
3long status = (long)stack;
4if(status < 0)
5  return status;

It's a good idea to mark a page at the beginning of the stack as non-readable and non-writable to protect any valid adjecent memory if there is any. Stepping over the guard page will get us a familiar segmentation fault. Why at the beginning and not at the end? Because the stack grows downwards, i.e. from a high address towards a lower one. Go here for more details.

1status = SYSCALL3(__NR_mprotect, stack, EXEC_PAGESIZE, PROT_NONE);
2if(status < 0) {
3  tbmunmap(stack, attr->stack_size);
4  return status;

We finally have all that is needed to spawn a new kernel task sharing with the main task everything that you would expect threads running within the same process to share. The man page of clone discusses all of the parameters in details.

2flags |= CLONE_THREAD;
3int tid = tbclone(start_thread, *thread, flags, stack+attr->stack_size);
4if(tid < 0) {
5  tbmunmap(stack, attr->stack_size);
6  free(*thread);
7  return tid;

Note that the user function is not called directly and start_thread is called instead. It does some internal book-keeping and eventually calls the user-supplied function:

1tbthread_t th = (tbthread_t)arg;
2uint32_t stack_size = th->stack_size;
3void *stack = th->stack;

When the function returns, the stack memory needs to be freed (munmaped) and the thread terminated. The clone function took some provisions for calling sys_exit with the status returned by the wrapper. However, here we remove the stack from underneath our feet, and we cannot risk any C function write over it. Therefore, we need to call sys_munmap and sys_exit in one piece of assembly.

 1register long a1 asm("rdi") = (long)stack;
 2register long a2 asm("rsi") = stack_size;
 3asm volatile(
 4  "syscall\n\t"
 5  "movq $60, %%rax\n\t" // 60 = __NR_exit
 6  "movq $0, %%rdi\n\t"
 7  "syscall"
 8  :
 9  : "a" (__NR_munmap), "r" (a1), "r" (a2)
10  : "memory", "cc", "r11", "cx");
11return 0;

The full code.

Lessons learned

  • The logic involved in running threads does not seem to be horrendously complicated, at least not yet.
  • Glibc is quite convoluted and figuring out what is going on and where is a challenge. It is probably justified by the range of platforms it supports, but the documentation could be better.
  • Threading is tightly coupled with glibc in the areas that I have not initially suspected: the dynamic linker to support TLS, globals in TLS (errno), locales, finalizers, and so on.
  • POSIX threads have an interesting relationship with the Linux task model. This document describes the initial incompatibilities that have ultimately been ironed out. It is a very interesting read if you want to understand how the userspace and the kernelspace interact to implement a threading system.
  • Ultimately all this is not necessarily a black magic. At least not so far.