The main idea of the streaming algorithm is to load a large chunk of data from slow memory into fast memory, use it as much as it can to achieve high arithmetic intensity, and then repeat with the next chunk of data. By accessing data in a contiguous manner, we reduce the amount of random disk accesses we would otherwise do when accesses nodes and edges, which may be distributed anywhere in storage. We see on the next slide how we can access data for a graph with this streaming approach, by sharding the graph so that we can access contiguous chunks of a subgraph and improve performance.
The main idea of the streaming algorithm is to load a large chunk of data from slow memory into fast memory, use it as much as it can to achieve high arithmetic intensity, and then repeat with the next chunk of data. By accessing data in a contiguous manner, we reduce the amount of random disk accesses we would otherwise do when accesses nodes and edges, which may be distributed anywhere in storage. We see on the next slide how we can access data for a graph with this streaming approach, by sharding the graph so that we can access contiguous chunks of a subgraph and improve performance.