So are the parameter values in the parameter servers not identical? In this case won't we be creating some sort of error in going from one parameter server to 2 with different values.
mperron
@ote In this model the parameters are sharded over the parameter servers. This means that the sets of parameters each one holds is disjoint, so you dont have an issue of the same parameter with different values (on the parameter servers).
Lawliet
So the amount of communication now doubles since a node has to talk to two parameter servers, but the amount of data each server has to transmit is now half because the parameters are split across the nodes.
So are the parameter values in the parameter servers not identical? In this case won't we be creating some sort of error in going from one parameter server to 2 with different values.
@ote In this model the parameters are sharded over the parameter servers. This means that the sets of parameters each one holds is disjoint, so you dont have an issue of the same parameter with different values (on the parameter servers).
So the amount of communication now doubles since a node has to talk to two parameter servers, but the amount of data each server has to transmit is now half because the parameters are split across the nodes.