It's worth mentioning that ASICs for neural networks are emerging. The following is a great series of paper:
And the authors are already starting their companies.
ASICs are designed for a certain purpose and are thus much more efficient when compared to more general purpose processing units. When looking at Bitcoins, ASICs were quickly developed after CPUs and GPUs were seen as too slow and unoptimized. After CPUs, GPUs were able to bump up the hashing power but only part of the GPU was being used for these calculations and a lot of energy were being spent in heat. ASICs were developed to become mining machines and have improved the cost from ~12 MH/s per dollar to >1660 MH/s per dollar.