As a follow-up on the matrix solver, NVIDIA has developed cuSPARSE library for dealing with sparse matrices, which offers a significant speedup compared to other approaches.

More details can be found here: https://developer.nvidia.com/cusparse