When we preform an asynchronous send, we need to commit to the kernel that we won't be changing the memory, so that the kernel can be sure that whenever it is ready to send the data, it will have access to the memory contents. Another possible way to implement this would have been to immediately copy the memory contents to kernel space, but we will run out of kernel resources too fast with this approach.

Question: What incorrect behavior might happen if a piece of code looked like this:

const int N = 1024;
float* data = new float[N];
//
// fill in data here...
//
Handle h = async_send(data, sizeof(float) * N);
delete [] data;
while (!check_send(h)) { }     // check_send return true if send is complete

printf("Message has been sent!\n");

Here, the program is deallocating data before confirming that the message has been sent. Therefore, it is highly likely for the asynchronous send to fail. To fix this, delete data after coming out of the check_send loop.

I have a question about CHECKSEND and CHECKRECV: is the only way to implement those a spinning loop? I always thought spinning in a loop was wasteful and costly (is this true?). But I really can't think of another way to continuously check if the message has been sent/received.

@taz: How about interrupt routine which interrupts when the sending or receiving has been completed?