Please see Fast Algorithms for Convolutional Neural Networks by Lavin and Gray for a detailed treatment of work-efficient convnet layer implementation.