A relevant example of memory constraints on scaling occurred in assignment 2. When attempting the scale the solution up to large numbers of circles, not all of the data could fit in shared memory at the same time. The solution is either to get more memory or to shrink the amount of memory in the working set at any given time.
grizt
Google recently published an article in one of its engineering blogs discussing how running tests at very large scales (on the PB level) exposes bugs not found otherwise. In one of its latest experiments, it had to scale back to "only" 50PB because that's all it could fit in storage within reasonable communication of the cluster.
A relevant example of memory constraints on scaling occurred in assignment 2. When attempting the scale the solution up to large numbers of circles, not all of the data could fit in shared memory at the same time. The solution is either to get more memory or to shrink the amount of memory in the working set at any given time.
Google recently published an article in one of its engineering blogs discussing how running tests at very large scales (on the PB level) exposes bugs not found otherwise. In one of its latest experiments, it had to scale back to "only" 50PB because that's all it could fit in storage within reasonable communication of the cluster.