Some applications of memory-constrained scaling:
Data processing for analytics, machine learning, scientific computing.
dasteere
Can this concept be applied to streaming algorithms as well? This would allow us to process "infinite" memory, although the maximum speedup would be reduced.
Abandon
Still confused about the difference between memory constrianed scaling and time constrained scaling. They all compare the work done in the same time. How can the memory constraint be indicated by the work per unit time?
jkorn
@Abandon, with time-constrained scaling, you want to do more work in the same exact amount of time, so you just measure the total amount of work done on P processors vs just the amount of work done on 1 processor in the same exact amount of time. This is your speedup. I guess this is the most intuitive one to think about.
With memory-constrained scaling, you are assuming that moving from 1 processor to P processors gives you P times more memory to use. You aren't looking at the work done in a fixed amount of time, or solving an identical problem faster, but rather you're interested in how much faster you can solve problems of a larger size. Across multiple processors you have more total memory to use, so you can manage larger problem sizes. And so in this case, your speedup is determined by the rate at which you get work done on your P processor setup vs just with 1 processor, since this tells you how much more performance you are getting from your parallel setup. When we are looking at the speedup for time-constrained scaling, the time is fixed, but here the time may vary; and with problem-constrained scaling, the total work is exactly the same, so you just compare times - but here, the overall work varies as well, which motivates the comparison of rates.
chenh1
The formula above is actually (work(P processor)/time(P processor)/(work(1 processor)/ time(1 processor)), and we can transform to work(P processor) * time(1 processor)/ time(P processor) * work(1 processor)
POTUS
Strong scaling, as opposed to weak scaling, refers to keeping the problem size the same and adding more processors.
lfragago
The trick to calculate the memory constrained scaling is to fix the work done by every processor to be the same as the baseline. Then we re-scale the problem size as:
Work_done_per_processor x P (where P is the variable number of processors)
Some applications of memory-constrained scaling: Data processing for analytics, machine learning, scientific computing.
Can this concept be applied to streaming algorithms as well? This would allow us to process "infinite" memory, although the maximum speedup would be reduced.
Still confused about the difference between memory constrianed scaling and time constrained scaling. They all compare the work done in the same time. How can the memory constraint be indicated by the work per unit time?
@Abandon, with time-constrained scaling, you want to do more work in the same exact amount of time, so you just measure the total amount of work done on P processors vs just the amount of work done on 1 processor in the same exact amount of time. This is your speedup. I guess this is the most intuitive one to think about.
With memory-constrained scaling, you are assuming that moving from 1 processor to P processors gives you P times more memory to use. You aren't looking at the work done in a fixed amount of time, or solving an identical problem faster, but rather you're interested in how much faster you can solve problems of a larger size. Across multiple processors you have more total memory to use, so you can manage larger problem sizes. And so in this case, your speedup is determined by the rate at which you get work done on your P processor setup vs just with 1 processor, since this tells you how much more performance you are getting from your parallel setup. When we are looking at the speedup for time-constrained scaling, the time is fixed, but here the time may vary; and with problem-constrained scaling, the total work is exactly the same, so you just compare times - but here, the overall work varies as well, which motivates the comparison of rates.
The formula above is actually (work(P processor)/time(P processor)/(work(1 processor)/ time(1 processor)), and we can transform to work(P processor) * time(1 processor)/ time(P processor) * work(1 processor)
Strong scaling, as opposed to weak scaling, refers to keeping the problem size the same and adding more processors.
The trick to calculate the memory constrained scaling is to fix the work done by every processor to be the same as the baseline. Then we re-scale the problem size as:
Work_done_per_processor x P (where P is the variable number of processors)