GPU's are built this way because they typically are used in Graphics processing (which is inherently parallel), and in ML matrix operations (which are also parallel).