In case of the incorrect ISPC solution where 'sum' is uniform variable that is shared by all the program instances. I was wondering if would be possible for the ISPC library to determine that programs such as these, where 'sum' is used incorrectly and it automatically can determine to use a local variable to store the partial sum and return the sum of the partial sum as the result to caller ?

Considering that the entire program is stalled until all the program instances complete their execution, I believe this could be done by the ISPC library. This will lead to a simpler code for the programmer.

Actually the program at left does not compile. It would generate a compile-time type error of assigning a non-uniform value to a uniform one. The type system provides the safety in this case.

A design point of ISPC is to provide a very low level of abstraction so the programmer can maintain a good picture of the implementation of the abstractions on the machine. (Performance is predictable.) Converting a uniform value to a non-uniform one would not be in the spirit of performance transparency.

I have a question about the 'foreach' language construct. From what I understand, it's basically a way of telling the compiler that each iteration of the loop will be independent of any other.

However, the 'foreach' loop that calculates the partial sum in the program on the right doesn't seem to satisfy that property. Each iteration of the loop uses the previously calculated value of the partial sum to calculate the new partial sum value. I've thought about it for a bit, and I understand that the code on the right still does work, since each instance in the gang is calculating it's own partial sum in an appropriate manner, by using it's own non-uniform variable 'partial'.

So, would it be more accurate in that case to say that the 'foreach' construct specifies that loop iterations carried out by separate instances of a gang will be independent from one another, but loop iterations that are performed by a single instance in a gang may be dependent on previous iterations executed by the same instance?

@TypicalChazz: Good point! While ISPC foreach does indeed mean run these iterations in whatever order and on whatever instance you want, iterations distributed to the same instance will be able to observe changes to instance-local variables from prior iterations.

However, (and someone should double check this through a careful read of the spec) I don't believe ISPC places any requirements on the order in which an instance performs the iterations that are distributed to it as a result of a foreach. In the example above, that's okay, since computing a partial sum is associative and commutative.

@kayvonf: Confirmed that ISPC does not place any requirement on the order in which an instance performs iterations distributed to it by foreach. Following is an example from ISPC documentation which exhibits undefined behavior:

uniform float a[10][100];

foreach (i = 0 ... 10, j = 0 ... 100) {

    if (i == 0)

        a[i][j] = j;

    else

        // Error: can't assume that a[i-1][j] has been set yet

        a[i][j] = a[i-1][j];

}

Thanks @afa4! Great work.

In lecture, Kayvon said that each instance runs the code within the sumall2 function, which implies that each instance is also calling reduce_add() and setting sum to equal its return value. But that would mean that all instances write to the sum variable at the same time which causes a compile time error.

Can someone please explain what's really going on when we call reduce_add() within an ISPC function?

@Kapteyn. The semantics of reduce_add define away the problem you mention. It is a cross-instance built-in function that ensures that the accumulation performed by each instance into the single uniform variable is atomic.

https://ispc.github.io/ispc.html#reductions