not sure why functional structure would hinder performance? from previous slides, my understanding is functional structure can execute side-effect-free programs in parallel, such that we can avoid the overhead for communication.. where is the bottleneck of this model that I missed?
arcticx
@yangwu I think the last bullet point has nothing to do with performance. Instead, using functional structures (i.e. write map function explicitly) is less flexible and natural than writing imperative codes (use foreach and let the compiler do the translation for you). That's why functional languages in general are not as popular as imperative languages. To facilitate or encourage programmers to write high-performance codes, the above mentioned programming languages adopt the imperative syntax that most programmers are familiar with rather than functional syntax which is more obscure to many programmers.
yikesaiting
Speaking of side-free function. ISPC functions are not totally independent. It needs to know at least how wide is the SIMD vectors.
grizt
To go further on the functional model of parallelism, I have a question about a slightly more complicated algorithm and how it might be parallelized using methods we've discussed. Say we have a function that adds an array of numbers together called sum. We implement sum recursively using divide and conquer: take the left and right halves, sum them in parallel, then add the sums from each half together. (In 15210, we learn that this takes approximately log(len(array)) iterations.) We could conceivably count the number of threads needed at the beginning based on the length of the array, but it still doesn't fit into our map model. Is there syntax in ISPC for this sort of thing? Or is this kind of parallelism not as practical?
bysreg
Do we have data-parallel model enforced programming language? or is it just a theory?
vincom2
@grizt I think ISPC has reducers? (I would try and link to github or something, but THE GREAT GITHUB OUTAGE OF 2016)
CaptainBlueBear
@yangwu I think the last bullet is referencing what was mentioned 3 slides back as the drawback of the data-parallel model. It gets more and more complicated to describe complex data flows (aka high-performance implementations) because we're limited on what operations we can do by our library of operators
PIC
I see how a data parallel model could be useful for massive amounts of data, use cases such as Hadoop. But does the cost of starting new threads outway the costs of actually executing map, reduce, fold, etc in parallel?
not sure why functional structure would hinder performance? from previous slides, my understanding is functional structure can execute side-effect-free programs in parallel, such that we can avoid the overhead for communication.. where is the bottleneck of this model that I missed?
@yangwu I think the last bullet point has nothing to do with performance. Instead, using functional structures (i.e. write
mapfunction explicitly) is less flexible and natural than writing imperative codes (useforeachand let the compiler do the translation for you). That's why functional languages in general are not as popular as imperative languages. To facilitate or encourage programmers to write high-performance codes, the above mentioned programming languages adopt the imperative syntax that most programmers are familiar with rather than functional syntax which is more obscure to many programmers.Speaking of side-free function. ISPC functions are not totally independent. It needs to know at least how wide is the SIMD vectors.
To go further on the functional model of parallelism, I have a question about a slightly more complicated algorithm and how it might be parallelized using methods we've discussed. Say we have a function that adds an array of numbers together called sum. We implement sum recursively using divide and conquer: take the left and right halves, sum them in parallel, then add the sums from each half together. (In 15210, we learn that this takes approximately log(len(array)) iterations.) We could conceivably count the number of threads needed at the beginning based on the length of the array, but it still doesn't fit into our map model. Is there syntax in ISPC for this sort of thing? Or is this kind of parallelism not as practical?
Do we have data-parallel model enforced programming language? or is it just a theory?
@grizt I think ISPC has reducers? (I would try and link to github or something, but THE GREAT GITHUB OUTAGE OF 2016)
@yangwu I think the last bullet is referencing what was mentioned 3 slides back as the drawback of the data-parallel model. It gets more and more complicated to describe complex data flows (aka high-performance implementations) because we're limited on what operations we can do by our library of operators
I see how a data parallel model could be useful for massive amounts of data, use cases such as Hadoop. But does the cost of starting new threads outway the costs of actually executing map, reduce, fold, etc in parallel?