how exactly is the update done from the result of other nodes?
Firephinx
@sherwood What do you mean? I think in this example, each node starts out with the same copies of all the parameters. Then they each do separate work with their images and update their parameter gradients locally. Afterwards, they send their local parameter gradients to a parameter server that updates the parameter values using the gradients from all the other nodes. Then the parameter server sends every other node the new parameter weights and then each node continues working on another set of images.
how exactly is the update done from the result of other nodes?
@sherwood What do you mean? I think in this example, each node starts out with the same copies of all the parameters. Then they each do separate work with their images and update their parameter gradients locally. Afterwards, they send their local parameter gradients to a parameter server that updates the parameter values using the gradients from all the other nodes. Then the parameter server sends every other node the new parameter weights and then each node continues working on another set of images.