Why is the formula of total communication cost equal to num_message timing (communication time - overlap)?
ESINNG
Because, when doing communication, the system can also do computation. So some of the time in communication is hidded by computation. And the rest is really what matters for the cost.
rokislt10
Another way of thinking about, in economic terms, is the opportunity cost of communication. If part of the time that you spend communicating is time that you would be spending on computation anyways, then it's not really part of the opportunity cost, is it? However, if time is spent on communication and no computation time can be spent during the waiting time, then that is a true opportunity cost.
srw
It seems like an easier way of thinking about this is, the communication cost of one message is the amount of the communication that occurs serially (is not concurrent w/ other tasks).
Thinking about it this way, it is also easy to see how much communication cost will slow down a program. Hint: Amdahl's law.
yuel1
Another great thing to think about is arithmetic density. In the equation above, the larger overlap becomes, the less our total communication cost becomes, which is what Kayvon meant by we get computation for free if our program is memory bound.
Why is the formula of total communication cost equal to num_message timing (communication time - overlap)?
Because, when doing communication, the system can also do computation. So some of the time in communication is hidded by computation. And the rest is really what matters for the cost.
Another way of thinking about, in economic terms, is the opportunity cost of communication. If part of the time that you spend communicating is time that you would be spending on computation anyways, then it's not really part of the opportunity cost, is it? However, if time is spent on communication and no computation time can be spent during the waiting time, then that is a true opportunity cost.
It seems like an easier way of thinking about this is, the communication cost of one message is the amount of the communication that occurs serially (is not concurrent w/ other tasks).
Thinking about it this way, it is also easy to see how much communication cost will slow down a program. Hint: Amdahl's law.
Another great thing to think about is arithmetic density. In the equation above, the larger overlap becomes, the less our total communication cost becomes, which is what Kayvon meant by we get computation for free if our program is memory bound.