A good reference gaining a basic understanding of deep networks from an ML perspective (at least the forms commonly used for computer vision tasks) is Stanford's course CS231n: Convolutional Neural Networks for Visual Recognition. It's a fantastic introduction and highly recommended.
This reminds me of another good example in the field of ML which is the perceptron algorithm. If you're interested, look at notes from 15-451 http://www.cs.cmu.edu/~15451/lectures/lec17.pdf.
@Calloc I am familiar with the perceptron algorithm from 451, but I do not see the overlap between perceptron and deep learning.
@ote Each unit in a neural net is a linear combination of the inputs(Perceptron-ish) with some non-linear transformation to produce the output. A neural network is just a network of these units.
@ote A common tool in machine learning is the multilayer perceptron which is essentially a deep neural network. You can learn more about it here: https://en.wikipedia.org/wiki/Multilayer_perceptron