I'm a little confused by this slide, mainly the "this use of atomics does not impact ..." portion. Is the reason that the two blocks both doing an atomic add at the same time is safe because of the abstraction we make that guarantees that atomicAdd is actually atomic and will have no side-effects in this case?

@vrazdan It means that you can't know for sure which block will increment the value first. The updates will both happen (so there are no correctness issues here), but just because there is an atomic (instead of an unsafe increment) doesn't mean that one block will update the value before the other. The scheduler is not affected by the use of the atomic primitive.

Just to add on top of what @arjunh mentioned. If you are interested. Look in the intel manual for the instruction xadd. There are variants of this such as xchg, etc. These are atomic instructions which the CPU guarantees atomicity.

There is also the "lock" keyword in Intel which achieves a similar function. Here's the 410 link to the intel isr: https://www.cs.cmu.edu/~410/doc/intel-isr.pdf

It may be good to emphasize that atomic does not mean thread safe. Here is a stack overflow question on atomic vs thread safe: http://stackoverflow.com/questions/21098494/atomic-properties-vs-thread-safe-in-objective-c

Although the question's focus is on Objective C, I believe the case is similar for CUDA. Someone correct me if i'm wrong.