Interesting comment in class: At the end of lecture, Kayvon mentioned how systems knowledge and ML knowledge can be combined in strategies like accounting for latency of operations in the momentum to create efficient, parallel ML algorithms :)
kayvonf
@BestBunny: This is the paper.
Asynchrony begets Momentum, with an Application to Deep Learning. Mitliagkas et al. 2016
An interesting feature of the problem domain is that "good-enough" approximations are completely valid and do relatively little to impact results. This gives researchers a lot of latitude to experiment.
vasua
At some point, there's only so much systems folk can do to speed up a given workload without fundamentally changing it. One example of changing the algorithm to add performance, rather than simply expecting frameworks to get faster, is the following DeepMind paper, which attempts to predict gradients and moves forward with the predicted values. This lessens the delay in waiting for the backpropagation to complete.
Interesting comment in class: At the end of lecture, Kayvon mentioned how systems knowledge and ML knowledge can be combined in strategies like accounting for latency of operations in the momentum to create efficient, parallel ML algorithms :)
@BestBunny: This is the paper.
Asynchrony begets Momentum, with an Application to Deep Learning. Mitliagkas et al. 2016
https://arxiv.org/abs/1605.09774
An interesting feature of the problem domain is that "good-enough" approximations are completely valid and do relatively little to impact results. This gives researchers a lot of latitude to experiment.
At some point, there's only so much systems folk can do to speed up a given workload without fundamentally changing it. One example of changing the algorithm to add performance, rather than simply expecting frameworks to get faster, is the following DeepMind paper, which attempts to predict gradients and moves forward with the predicted values. This lessens the delay in waiting for the backpropagation to complete.
https://arxiv.org/abs/1608.05343