How exactly does register checkpointing work? An initial Google search didn't turn much up. Is there another name for this?
If you are interested in reading more about the hardware aspect of hardware-supported transactional memory, I know a good reference: Harris, T., Larus, J. R., & Rajwar, R. (2010). Transactional memory (2nd ed.). San Rafael, Calif. (1537 Fourth Street, San Rafael, CA 94901 USA): Morgan & Claypool.
Its e-book is available in CMU Library at: http://vufind.library.cmu.edu/vufind/Record/1441870
Specifically, Chapter Five discusses different proposed design of HTM, including aspects of register checkpointing.
What happens if the OS swaps a thread out? Does HTM try to ensure that this does not happen in the middle of a transaction?
I suspect that the transaction is simply aborted, but is there some other logic?
The transaction would definitely abort if the processor receives an interrupt in the middle of the transaction.
A nice discussion is here: http://www.realworldtech.com/haswell-tm/2/
Under what work loads is HTM faster than lockless approaches?