Cost of this approach: longer time to converge (perhaps 10% more iterations)
Benefit of this approach: no time wasted sitting idle, waiting for updated parameters
o_o
This approach is helpful for this situation because if all the nodes had to communicate with each other the updated values of the parameters, there would be constant contention on the parameter values as all nodes try to update them, and then that update has to propagate to all the other nodes. This way, nodes are able to update the parameters when they can, and go on using a potentially older version of the parameters. They will eventually get the updated ones so they will still converge, it will just take longer.
Cost of this approach: longer time to converge (perhaps 10% more iterations) Benefit of this approach: no time wasted sitting idle, waiting for updated parameters
This approach is helpful for this situation because if all the nodes had to communicate with each other the updated values of the parameters, there would be constant contention on the parameter values as all nodes try to update them, and then that update has to propagate to all the other nodes. This way, nodes are able to update the parameters when they can, and go on using a potentially older version of the parameters. They will eventually get the updated ones so they will still converge, it will just take longer.