Seeing volatile in action

I had the fortune of working at a company which worked with medical devices during my summer internship which was right up my alley due to my immense interest in firmware engineering and embedded systems. I had a great colleague who was already working there and I got to work hands-on on some major issues. One of them was very interesting and I think it warrants a story which will be covered here. To keep the background short, we were working to eliminate BLE packet loss and after quite some research and experimentation, my colleague suggested that we look into upgrading our SDK to the latest one supported by the chipset we were working with, which introduced BLE 4.2, which in turn allowed us to use this feature called Data Length Extension which allowed us to extend our Protocol Data Unit (PDU) size from 27 bytes to 251 bytes. The reasoning behind this will be covered some other time!

The first approach we tried was to simply replace the necessary header files and just get it to compile without any errors and directly flash it onto the board to see if it worked. As expected, this did not work, but it was just a shot into the void with a miniscule chance of working. The newer SDK was released around 8 years after the SDK we were currently using, so expecting it to work is like believing in magic. So we took the next sensible step, took an example from the latest SDK which was similar to our firmware, and got around to porting it part by part keeping in track what changes were made and ensuring it compiled at each step. And we got it to a point where the connection and disconnection behavior was the same as the other devices with the older SDK. Or so we thought!. There was a very strange issue we encountered. The disconnect functionality did not work as expected using the actual button on the device which was supposed to trigger it, and neither did the buzzer work (but this was a different issue altogether). The disconnect functionality basically relied on a global boolean variable called disconnect. Very horrible coding standards, yes. But this simple approach was enough for our use case. Time to dig deeper.

Compared to general purpose operating system, RTOS (Real Time Operating System) operates very differently. It is very minimal and much simpler compared to consumer oriented operating systems like Linux. But RTOS reigns supreme in the world of embedded systems because its point isn't to provide extensive security and a lot of features but to provide a very thin layer above the hardware so that the firmware engineer working with a board does not need to implement basic things like schedulers, delay loops, synchronization primitives, etc. But one of the most important features of an RTOS is scheduling. In fact there are classifications of RTOSes based on how they schedule tasks and what kind of time constraints they operate under. To keep it simple, RTOSes on single core processors employ concurrency in order to give the illusion of many tasks running at the same time instead of true parallelism where different hardware cores handle different tasks. This is done by rapidly switching through multiple tasks, (horrible visualization given below).

Modern compilers make tons of optimizations in order to make well written and readable code as fast as possible so that programmers do not need to resort to weird black magic to write performant code. This is partially true though because there are still some cases where hand-rolled assembly, specific bitwise operations instead of arithmetic operations, etc. are still required but for most cases, compiler optimizations are very powerful. A very common optimization and the one relevant to this article is moving a variable which is commonly accessed within a loop, to a single access outside the loop (into a register) in order to save some cpu cycles by avoiding unnecessary memory accessess, provided the value does not change during the loop. This is probably better illustrated by the pseudo-assembly example given below:

loop:
ldr r0, [addr+8] ; load value at memory [addr + 8] into register r0
	cmp r0, #some_literal ; compare the value in r0 to some_literal
	be loop ; if r0 == some_literal jump to `loop` else go to next instruction

vs.

ldr r0, [addr+8] ; load value at memory [addr + 8] into register r0
loop:
	cmp r0, #some_literal	; compare the value in r0 to some_literal
	be loop ; if r0 == some_literal jump to `loop` else go to next instruction

But of course, this would only make sense if we can safely assume that the variable that is loaded into r0 is not changed during the execution of the loop.

In modern systems, computers have a CPU, some RAM (memory), and some ROM (storage) and a lot of other peripherals. Typically for most programs, files are read from ROM into RAM and other variables that are defined within the program (stack or heap) are loaded from the RAM into the CPU cache and then into the CPU register. The CPU has very basic memory units called registers which is where data is stored before performing arithmetic, bitwise and other operations on the data. So obviously, the latency between fetching variables from RAM, loading it to the cache and then performing some operation on it is much higher than performing operations on variables that are already present in the cache or registers. The access speed goes like this: ROM -> RAM -> CPU cache -> CPU register, but the actual difference in time is several orders of magnitude faster/slower. So obviously, the compiler prefers to pre-load variables into the register and use those as much as possible instead of fresh memory accesses every single time.

The C programming language is truly remarkable given its simplicity and focus on just providing the bare minimum needed to incorporate basic english vocabulary when writing computer programs compared to modern languages which focus on memory-safety, ease of use, etc. This is also one of the reasons it is still the go-to language for embedded systems because the grammar of the language is so simple that writing a compiler for a custom platform has been done many times before and is well documented and there are entire tools for this, and since there are also practically very few abstractions, C is extremely fast and performant due to being able to produce binaries with very little overhead, which is perfect for environments which dont have a lot of processing power to be spent on more modern features. Keeping this in mind C just provides essential keywords for how variables should be treated and stored, like static, const, and here comes our point of interest, volatile.

Given all the above context must've given you an idea of where this is all going. So coming back to the problem at hand with the device not disconnecting as needed, the code was layed out like this:

// includes
bool disconnect = 0;
void task1(...) {
	...
	while (1) {
		...
		if (disconnect) {
			// do the actual disconnecting stuff
		}
		...
	}
}

void task2(...) {
	...
	while (1) {
		...
	}
	...
}

so in task1() the if (disconnect) { ... } seemed to have absolutely no effect. We ran it through the debugger and it did not even seem to step into the if block even though disconnect should technically have been true at that point of execution. We even looked at the disassembly and tracked the instruction pointer to see how the branching occured because the disassembly had the branch but it never actually branched! But for some reason it worked when we intentionally introduced a similar statement in task2(). It took quite some time to figure out what was even going on. and then it struck me! All the random videos I've seen on youtube and articles I've read about the glorious keyword: volatile. Making disconnect as a volatile variable somehow just worked magically. And thats when I understood the situation and what could have possibly happened, and this is kind of when all the content I'd come across about volatile made so much sense to me, because before this I never got to use it in a meaningful way because most of my code never encountered issues like this.

And finally we circle back to repeating the same mistake I've seen before, writing yet another article about volatile but this time it is different. I only understood the existence of this keyword when I personally had an issue tied to optimizations and caching and how volatile helps to avoid these kinds of scenarios. So essentially, volatile instructs the compiler to never perform any kind of optimizations related to memory accesses on a variable with this keyword attached to it and basically keeps the compiler alert to the accesses of this variable by signifying that there can be unexpected changes to this variable so the compiler must make no assumptions about the value of the variable and must always access memory directly to get the value of this variable regardless of whether this variable is in a register or not. So we finally concluded from our debugging sessions and some thinking that what most likely happened was the compiler just assumed that disconnect could not change inside the while (1) because there is nothing actively modifying disconnect there, and had it stored in a register outside the loop and only bothered to access the same version which was not updated to the actual in-memory value of disconnect and caused the branch to never execute because it was repeatedly reading the old value which was false.

The intersection of all the topics I'd learnt but never got to practically use was actually just beautiful, because it combined topics from operating systems, computer architecture, compiler design and C. Its always great when you use some forbidden knowledge you've been hoarding for some time without using it and it suddenly comes in handy to fix an issue. One instance that allowed me to combine all these topics to narrow down the issue! Of course, there's always the fact that I can probably spot something like this happening much faster the next time it happens simply due to the experience of already solving it once, but it still felt great nonetheless. And of course finally after some research, its never a good idea to rely on volatile to manage problems like this which typically occurs due to globals being shared by concurrent or parallel tasks. Always prefer using proper synchronization primitives instead of fixes like volatile which isn't really a good fix in the first place, but again the use case here was very simple. It also taught me to always have a mental model of how everything is viewed by the compiler, the operating system and that what happens during runtime isn't always mapped out directly to the code you've written, or to put it simply, as a programmer it was easy for me to just assume that nothing could go wrong with disconnect in this case because its just two tasks that are running through RTOS and they're sharing a variable but for the compiler, both the "tasks" are just functions isolated from each other (because one does not depend on the other in our codebase) and all it sees, is the current execution path and not the full execution of variables and functions, which is what caused this issue in the first place.