1. Diagrams that help visualize what's happening. I prefer a hybrid of sequence, timeline and activity diagrams. I don't have a ready example at hand, but [1] comes close to what I use. YMMV though - I know some team members in my past jobs preferred state transitions to these, because they found them clearer to think through.

2. A technique we used during team reviews was borrowed from the book "Debugging Applications" by John Robbins [2]. Although the book was about debugging Windows applications, it had good tips on general software engineering and debugging, and I still recommend it now. One technique in it was the multithread simulation review - an excerpt:

The trick is to assign one person to each thread in your code and one person to each synchronization object. When you review the code, pretend that each thread is running at realtime priority on its own dedicated CPU. Each "thread person" walks through the code paying attention only to the particular code that his thread is supposed to be executing. When the "thread person" is ready to acquire a synchronization object, the "object person" literally moves behind the "thread person." When the "thread person" releases a synchronization object, the "object person" goes to a neutral corner of the room. In addition to the thread and object representatives, you should have some developers who are monitoring the overall thread activity so that they can assess the program's flow and help determine the points at which different threads deadlock.

Obviously, you need a number of like-minded and enthusiastic team members to carry it off.

[1]: http://community.qnx.com/sf/wiki/do/viewAttachment/projects....

[2]: https://www.amazon.com/Debugging-Applications-DV-MPS-Program...

For instance, in the Java ecosystem there are the various Executors; Executors aren't good at everything, but they are good at many cases that turn up, and I've frequently spent a few days trying to understand what was going on with "sophisticated" solutions in Scala and/or Clojure and then spent 20 minutes coding it up with an Executor and at that point had something that worked correctly all the time and used all the CPU cores to boot.

I'd agree with this! I'd go further and say in most languages avoid concurrency entirely if you can if it's not going to lead to measurable improvements. Concurrent code can be a minefield of problems and creates difficult to debug code so go with non-concurrent if it's all the same.

Of course the price of using a tool like Erlang [1] is that all that knowledge of mutexes and semaphores goes to waste and different concepts like immutability and let-it-crash must be understood if not mastered.

[1]: or perhaps Clojure or Go.

https://doc.rust-lang.org/book/second-edition/ch16-00-concur...

Part of being a good programmer involves trusting other developers. Sharing code, reusing code, etc. That's how our community works and trying to use basic principles in your code isn't always the way to go. To manage concurrent code today it's better to use a Queue for instance. Setup your queue and just push task into it. Synchronously or asynchronously, you get to pick what ever you want. It usually comes with a bunch of other features, like maintaining a certain order when asynchronous tasks get executed. In most of the modern languages, they offer thread safe and none thread safe data structures. Pick the one you need. And that's just one example.