This is one of the reasons why turning a sequential program into a parallel one is very difficult to automatize: sometimes the best parallel solution requires extra knowledge specific to the context of the computation you are doing.
On the same topic, parallelism-friendliness must be taken into consideration when you are designing an algorithm for computational mathematics. The algorithm with the better convergence speed might be executed slower on real machines that another algorithm with a worse convergence speed but better parallelism-friendliness.
tkli
This is quite interesting, and it seems to suggest that computer scientists/programmers have to collaborate and work together with scientists and mathematicians to develop algorithms that are parallelizable. In general, turning sequential programs into parallel ones seems to be highly nontrivial to the point of being impossible for computers to do at least in the near future simply because of "extra" knowledge needed. It would be interesting to learn about the cutting edge in the field and the directions that this will go.
paramecinm
The more interesting thing is that sometimes the parallel version of algorithm which is only an approximation of the original algorithm, may have better results. Moreover, there is often no theoretical guarantee. Like what is said about downpour SGD in this paper: 'There is little theoretical grounding for the safety of these operations for nonconvex
problems, but in practice we found relaxing consistency requirements to be remarkably effective.'
vasua
An article I was reading yesterday (discussion here), has some interesting thoughts about how modifying the intermediate representation of a compiler can make it easier / harder / impossible to detect automatic parallelism. MIT seems to be doing some work in this area with LLVM.
This is one of the reasons why turning a sequential program into a parallel one is very difficult to automatize: sometimes the best parallel solution requires extra knowledge specific to the context of the computation you are doing.
On the same topic, parallelism-friendliness must be taken into consideration when you are designing an algorithm for computational mathematics. The algorithm with the better convergence speed might be executed slower on real machines that another algorithm with a worse convergence speed but better parallelism-friendliness.
This is quite interesting, and it seems to suggest that computer scientists/programmers have to collaborate and work together with scientists and mathematicians to develop algorithms that are parallelizable. In general, turning sequential programs into parallel ones seems to be highly nontrivial to the point of being impossible for computers to do at least in the near future simply because of "extra" knowledge needed. It would be interesting to learn about the cutting edge in the field and the directions that this will go.
The more interesting thing is that sometimes the parallel version of algorithm which is only an approximation of the original algorithm, may have better results. Moreover, there is often no theoretical guarantee. Like what is said about downpour SGD in this paper: 'There is little theoretical grounding for the safety of these operations for nonconvex problems, but in practice we found relaxing consistency requirements to be remarkably effective.'
An article I was reading yesterday (discussion here), has some interesting thoughts about how modifying the intermediate representation of a compiler can make it easier / harder / impossible to detect automatic parallelism. MIT seems to be doing some work in this area with LLVM.