Multi-Threaded Coding Best Practices C++, NSFW Copyright © timmer 2016. All rights reserved. Version: 2016-09-13-1227 License: Permission to copy this work unaltered in its entirety including the copyright notification and license is granted universally in perpetuity for any non-commercial use. Charging a fee for the distribution of this work is considered a commercial use. [Illustration of NCP dancing past other NCPs tossing papers into the air. NCP reading paper. NCP catching falling paper.]

[The text style is matter of fact, dry, mostly humorless. It may reference the illustrations. But seems oblivious to the fact the illustrations include nude cartoon people (NCPs). With boobs, sex toys, etc. The NSFW level is naughty. But not X-rated. The art style is close to Andy Weir's Cheshire Crossing web comic. Equal numbers of males and females. Diverse skin color. Careful there's no correlation with skin color or gender with NCPs doing good or bad things.] [Notes: What are semaphores? You can't do anything without one. And it's destroyed as soon as you get one. Some options: viagra pills, cookies, candy bars, blow up dolls, condoms, ???, stop light, alarm clock, rooster.] [Thoughts on Illustrations: comic strip style – clean lines, simple coloring, minimal shading, exaggerated facial expressions, otherwise not much detail. Consistent body shape – kinda like the lego figures but without angles. Characters differentiated by skin color, hair color, hair style, breast shape. Probably more female character than male. People picked up the book cause boobies. And girl characters have all the drawing options of male characters plus breast shape. The most naughty bits are inferred from actions offscreen. None of the characters are recognizable as real figures. Though the second illustration for Rule #2 looks an awful lot like rick santorum. Wide range of skin colors – white, pink, brown, yellow, red, and everything in between. Maybe everyone has black hair and eyes to reduce number of options. Breast shapes http://www.stuff.co.nz/life-style/life/77894499/There-are-seven-types-of-breast-shapelingerie-brand-claims (though I can't tell the difference between bell and tear drop. Whatever, they're all beautiful.)

Characters are consistent across illustrations. Characters generally don't appear in more than one sequence. Except the final curtain call and the cover. No explicit genitals. But bare butts are good.]

Introduction – Why? - Background These best practices were developed during my time at OnLive. I didn't evangelize them then. The company had a large amount of code and hence inertia. I certainly didn't have the people skills to convince a large number of talented people they've been doing things wrong for their entire careers. OnLive went out of business for reasons unrelated to muti-threaded coding practices. [Illustration of NCP playing a video game with really long cables that go to a computer on the other side of the Cloud.]

CastAR hired me to replace the windows based software for their augmented reality glasses from their successful kickstarter campaign with software for entirely new hardware running on Android. I thought this would be a perfect opportunity to put these best practices into play. I was right, of course. And completely wrong. The first time I mentioned it, I got a face full of anger. I thought it would be pretty easy to convert a small number of very smart people working at a startup on a new idea. Wrong, wrong, and wrong. I tried mentioning it a few more times later to individuals. Still no traction. [Illustration of NCP on soap box with megaphone. Other NCPs doing the see/hear/speak no evil monkey thing.]

When CastAR and I parted ways, how to deal with multi-threaded issues was a contributing factor. I needed a new plan. A friend of mine wrote a book on ruby. No one ever questioned his credentials. Another friend of mine wrote a book which became a major motion picture. Obviously, writing a book is so easy a dork like me could do it. Besides, the next time someone gets all up in my face on this issue, I really just want to be able to say, I wrote the book on multi-threaded coding. [Illustration of NCPs yelling at another NCP holding and pointing at a book.]

Table of Contents TBD tk

Coding [Illustration of NCPs in hard hats with tools building a house. One is holding plans.]

Writing code is like building a house. The rooms are where the work is done. There are kitchens, dining rooms, living rooms, bedrooms, bathrooms, closets, garages, hallways, etc. A well-organized house makes it easy to get the work done. Everything you need to cook a meal is in the kitchen. The bed is in the bedroom. The dining room table is in the dining room. The same idea applies to all other rooms. This is what I will refer to as logical design throughout this document. [Illustration of a NCP working happily in the kitchen. Maybe other rooms.]

Imagine the chaos that would ensue in a house where this wasn't true. [Illustration of a NCP carrying a turkey from the oven, which is in the bedroom, to the dining room. The dining room table is in the garage.]

Threads If writing code is like building a house, then threads are the people that live and work in the house. When there's only one person living in the house, they can do whatever they want. They can leave dirty dishes in the sink. They can leave the toilet seat up. No one will care. They can eat the leftovers and no one will expect them to still be there. [Illustration of NCP in bachelor pad. Dirty dishes, pizza boxes, laundry piles (we'll ignore why naked people have piles of laundry), toilet seat up.]

But! The situation changes when the house is shared by many people. Suddenly you have to coordinate who gets to do what when. In the world of multi-threaded coding, this is called synchronization. [Illustrations of many (possibly) blind NCPs in one house. One is pulling a char out from another that's about to sit in it. Two ran smack into each other in the hallway. One just turned out the lights leaving another in the dark in the garage. One NCP just walked into the bathroom where another NCP is taking a shower. (we'll ignore why this would be a big deal to naked cartoon people.)]

Mutexes To synchronize threads, we need a thing that is unique – that there is only one of. Something a thread can take exclusive ownership of. Something that if I'm holding it, no one else can. You must wait for me to release it before you can access whatever the thing is protecting. That thing is a mutex. A mutex is a lock is a critical section. Different names. Same thing. [Illustration of a giant dildo with text: mutex, lock, criticalsection, dildo, sex toy.]

To further the house metaphor, a mutex is a key. You must acquire the key before you can go into the bathroom. While you're in there holding the key, anyone who tries the door will find it locked. They will need to either wait, or come back later. [Illustration sequence first NCP picking up dildo from table and going into a room. Second NPC finds the table empty and the door locked.]

Rule #1a: Minimize the time you hold a lock. The reasoning behind this rule is pretty obvious. When one thread is holding a lock, all other threads that want to use the protected resource must wait. It's surprising just how much this can affect performance of a system. One project I worked on held a lock while it inserted a video packet into a sorted list. On some systems we could play back at 15 fps. We changed the protected code to simply move the packet from one queue to another. Insertion happened while not holding the lock. Playback on that platform jumped to 40 fps! [Sequence of illustrations of NCP running back and forth waving a dildo, woo! While more and more NCPs show up and look unhappier about waiting. Clock on wall advances time.]

Rule #1b: Don't hold a lock any longer than you absolutely must. This rule is so important that it's worth repeating over and over again. You really don't want to hold a lock any longer than is absolutely necessary. Get in. Get out. Free the resource for the next person. [Illustration of NCPs waiting at empty table. Strange noises coming from locked door. Line of NCPs waiting. Clock on wall.] [Illustration of boy NCP staggering away from door. Dildo is now on table. NCPs look aghast. Girl NCP#2 nudges Girl NCP#1 who is trepidatiously reaching for dildo. Open door.]

Rule #1c: Treat every lock as if it's the only one in the entire universe. When you hold the lock, you stop everything in the universe. Planets don't spin. Supernova don't explode. Babies don't cry. Taxes don't get paid. No one breathes. The sun doesn't shine. [Illustration of NPC holding a lock on a spaceship over a dark frozen planet.]

Semaphores Suppose one thread, let's call him “thread”, puts data into a box. The box is protected by a mutex so another thread, let's call her “threadna”, can't pull incomplete and possibly corrupt data from the box. How does she know when to look in the box? [Illustration of NCP putting papers into a box. Another NCP waits.]

She could look continuously. But she must acquire the lock just to take a quick peek. Which makes it difficult for thread to acquire the lock and fill the box. [Illustration of NCP holding a dildo, shaking an empty box. A line of NCPs holding papers is waiting.]

She could poll i.e. look every so often. This might make sense if she has something useful to do for a while. [Illustration of NCP painting while another NCP puts papers in the box.]

But if she doesn't have anything useful to do she can set an alarm and go to sleep thereby freeing up CPU resources until they're actually needed. This alarm is called a semaphore – sometimes a signal. [Illustration of NCP walking away from a box with a squawking parrot on a dildo. Another NCP is rudely awakened.]

Multiple Threads Writing multiple threaded code is just like writing single threaded code except with locks and semaphores. NO! But that's the way I've seem many very talented, very experienced people write their code. It's maddening and saddening. We have covered in this document the tools that we need to keep our threads from crashing into each other. Progress. But the logical design pattern isn't good enough any more. We must also consider what the threads are doing. We have a beautiful logically organized house. That's the logical design. We also have people moving about inside the house. Imagine them leaving trails of breadcrumbs. I call this the thread diagram. Every place those trails intersect is a place the threads need to synchronize. For best performance, or even nominal performance, these opportunities for contention should be avoided. [Illustration of NCPs moving about the house with Family Circus style paths.]

Reduction Modern computers have been around for 50+ years. Single threaded coding best practices are well known. Multi-threaded coding adds a whole bunch of new pitfalls. Fortunately, we can borrow a concept from mathematics: reduce the problem (multi-threaded coding) to one you've already solved (single-threaded coding). We can treat each thread as if it were its own single-threaded process. We can pass data from one thread to another using well known inter-process communication (IPC) techniques. Problem solved! [Illustration of a row of houses each occupied by a single NCP. Separated by fences with gates. Each gate has a parrot and a dildo.]

That's actually a really good way to conceptualize the problem and solution. But you wouldn't want to implement anything that way. It's much more efficient to pass pointers to data instead of the data itself between threads. And obviously, threads can share common data that never changes. The critical concept here is to make the logical design and the thread diagram match. This is our version of the equivalence principle. In our house metaphor, each thread is assigned its own room (or suite of rooms). Between the rooms are containers for data. These containers are like chaperones that keep the threads apart. [Illustration of boy NCP and girl NCP separated by short old clothed chaperone holding out its hands to keep them apart. NCPs look twitterpated.]

Threads communicate with each other by putting data into containers and taking data out of other containers.

[Illustration sequence: boy NCP gives love note to chaperone who gives it to girl NCP surrounded by cartoon hearts.]

Separation of code and data is a very powerful technique. Conceptually, separate the parts of the code that perform actions from the parts of the code that hold data. But don't stop there. Literally separate them into different files in different directories. Put all of the code for a single thread in its own subdirectory. This code is single threaded. It doesn't use mutexes or semaphores. This code has no API other than what's needed to start and stop the thread. This dramatically reduces the complexity of the problem for the responsible individual. In practice, the shared data code is pretty small. I tend to put all container code in the same directory. Containers use mutexes and semaphores. Containers have a public API. Usually, they have two APIs – one to put data, the other get data. To review: Writing multiple threaded code is just like writing single threaded code except with locks and semaphores. YES! [Illustration of house with many rooms. One NCP per room. There are chaperones holding dildos standing on boxes with parrots on their heads between the rooms. Maybe in closets/hallways/airlocks.

Pitfalls Deadlocks A deadlock is when two threads block each other by holding a mutex the other is trying to acquire. [Illustration of two unhappy NCPs each holding a dildo away from the reaching hand of the other. Two empty tables. Two closed doors.]

An almost trivial solution is to never hold two locks at the same time. This works really well for a large number of real world problems – like data streaming. It fails when data is connected in complex ways – like a relational database. This documents concentrates on the former. The latter is beyond the scope – way, way beyond. [Illustration of an unhappy NCP holding two dildos, another NCP wearing a policeman's hat writing a ticket.]

Memory Allocation Rule #1 (a, b, and c) is to hold a lock for the shortest possible number of operations. There are a number of ways this rule can be broken without realizing it. The easiest is memory allocation. To allocate memory, you must grab a global lock and do a lengthy operation. Meanwhile, no other thread can allocate or delete memory. It's very much a stop the world scenario. Best practice is to allocate all the memory you need at initialization. [Illustration of caped NCP (holding a dildo) stopping the world like superman did by flying around it really fast to save Lois Lane.]

Pretty much every operation in the std c++ library allocates memory. I don't really want to say don't use them fancy newfangled STL code. But don't use a c++ construct (like a vector) if you can use something simple from C (like an array). Callbacks Sometimes a request for data can't be filled immediately. The caller can block and wait. Or it can go do something else until the response is available. Callbacks are a popular wait to deliver the response. It's a very easy method for the data provider to implement. Callbacks in single threaded code are just fine. But in the multi-threaded world, they are risky. They violate the equivalence principle - logical design and thread diagram are the same. The callback belongs to a certain logical block, “You”. But the code is executed from a different logical block, “Me”. Think of the parts as if they were your body. Do you really want any part of “My” body running amok inside “Your” body? [Illustration of pointy haired boss clown pointing to thread diagram. Seated boy and girl NCPs watch from foreground. One thread block looks vaguely like a butt. Another in a different but similar color looks looks vaguely like a penis aimed at or overlapping the butt.]

If callbacks are so dangerous, why is the technique ubiquitous? The problem is long term. The initial plan is usually pretty safe. Over time however, the project is touched by many hands. Some of those hands extend the function of the callback just a little bit. They test it. It works. Ship it. Repeat. Eventually the outside thread contamination reaches a place where maintainers have no idea they're working on code called from multiple threads. In the diabolical case, the test suite works fine. The problems only show up in the wild - a classic example of unintended consequences. [Illustration of girl NCP holding a positive pregnancy test.]

Sometimes you don't have a choice about what libraries to use. [Illustration of boy NCP waist up striking a super man pose hands on hips with big grin looking at girl NCP. Girl NCP is looking aghast at boy NCPs crotch. Clown holding riding crop.]

Best practice is to take appropriate precautions when a mandated library uses callbacks. Wrap the library with an API layer that doesn't use callbacks. Treat the callback as if it were its own thread – which it is. At the end of the day, the callback is producing data for the calling thread. Within the API wrapper layer, the quarantined callback simply puts the data into a container where the calling thread can extract it when needed. [Illustration of smiling girl NCP holding a condom. Boy NCP looks uncertain.]

Containers Containers simply hold data. The code for them should be pretty simple. I generally make a class with getters and putters that wrap primitive types. There really are only a small number of primitive types that should be used in containers. 1. Single Buffer 2. Double Buffer 3. FIFO 4. Most-Recent-Out 5. Write-Once-Read-Many Single Buffer works like an airlock. You put data in. I take data out. [Illustration of NCPs with space helmets (but not space suits) using an airlock.]

The problem with a single buffer shows up when filling it and draining it take a long time. You and I are blocked by the other's lengthy operation. The solution is to use a second buffer. I'm filling one while you're emptying the other. When we're both done, we swap buffers. [Illustration of NCP puts food into one airlock while another NCP puts trash into another. Then they swap. First NCP pulls trash from the second airlock while Second NCP pulls food from the first airlock.]

The problem with double buffers is when I'm done but you're not. I can wait. Or I can use a third buffer. The general case is to use a FIFO. These are well known from single-threaded coding. [Illustration of NCP giving newspapers to a line of dogs (FIDOs get it?). Another NCP takes the newspaper from the first dog in line. Dogs with no newspapers return to the first line to wait for a newspaper.]

Sometimes neither the producer nor the consumer can be blocked. At all. Ever. If the consumer falls behind, data must be discarded to keep the queue from getting full and blocking the producer. Three buffers are required to implement the Most-Recent-Out strategy. Originally, this was called a triple buffer but that term is often confused with a FIFO of size three. MRO buffers are marked empty, full, or in-use. The producer finds an empty buffer, fills it and marks it full. If there already was a full buffer, the old one is marked empty. The consumer finds the full buffer (if any), marks it in-use, uses it, marks it empty. [Illustration of NCPs using a MRO buffer.]

The last case we're going to cover in this document is the Write-Once-Read-Many buffer. Configuration data is mostly constant but it can change from time to time. Many threads need to read it so they know how to do what they do. We could broadcast it to each thread individually. Ick. It would be very disruptive to simply protect the data with a single mutex. Even worse. Fortunately there's a better way. Most of the time a reader can just read the data as if it never changes. But that could lead to data corruption if it's read while the writer is writing. The basic idea is to detect an error might have occurred and re-read the data. [Illustration of one NCP drawing a new picture. Many NCPs busily copying the last picture. One NCP's picture is messed up. It's a hybrid of the last picture and an even older picture.]

That leaves only the Lock Free Queue. These don't exist. Don't waste your time. I'm not even going to explain why. [Illustration of NCP selling unicorn hunting licenses.]

Summary Separate code and data. Isolate the code for each thread in its own directory. Thread code has no API, does not use mutexes or sempahores. Data container classes usually fit nicely in a single file. Containers have one or more external APIs. Containers do nothing other than hold data protected by mutexes and semaphores. Conclusion I hope you found the information in this document useful. And perhaps a little entertaining. Feel free to adopt the outlined techniques, use the associated source code, https://github.com/timmerov/aggiornamento, and evangelize. Please and thank you. [Illustration of waving NCPs, clowns, parrots, dildos, boxes, etc.]