As a matter of fact, asynchrony has been widely adopted by many modern Distributed Machine Learning frameworks, such as Microsoft Adam, the Parameter Server by Mu Li et al. Besides great speedup from relaxation of synchronisation, the accuracy of model can also be improved surprisingly.
jellybean
Any idea why the accuracy would improve? I find it counter-intuitive that relaxing synchronization improves accuracy. Maybe some level of randomness is beneficial?
As a matter of fact, asynchrony has been widely adopted by many modern Distributed Machine Learning frameworks, such as Microsoft Adam, the Parameter Server by Mu Li et al. Besides great speedup from relaxation of synchronisation, the accuracy of model can also be improved surprisingly.
Any idea why the accuracy would improve? I find it counter-intuitive that relaxing synchronization improves accuracy. Maybe some level of randomness is beneficial?