Define mappers, reducers and the data structures that's been processed using the provided framework.
(Optinally)Define how to partition the data in order to distribute the data into multiple machines.
System's responsibility:
Schedule mapper and reducer works in different worker machines. Perform the partition. Make sure the dataflow in the system is correct.
Khryl
For OpenGL:
Programmer's responsibility:
1. Specify how camera, lighting, transformation, texture, alpha_blending and all sorts of graphics setting are set up.
2. Specify list of primitives to be drawn in the scene and their properties like positions, normals and texture coordinates.
System's responsibility:
1. From programmer's input, figure out all the math for setting camera, lightning and so on.
2. Communicate with GPU to take advantage of its Graphics-specific units and parallelism to make rasterization more efficient.
kayvonf
@Khryl. I like your breakdown, but in modern OpenGL (e.g., OpenGL ES), there is no concept of a camera, lighting, or materials. This might seem unintuitive, but it's true!
hofstee
I'll take a stab at modern OpenGL:
Programmer's responsibility:
Provide input data for the system
Define how to process each incoming vertex
(optional) Define how much tessellation a specific patch gets
(optional) Define how to process each primitive
(optional) Define how to process each fragment generated by rasterization
(optional) Do something completely arbitrary, with user defined constraints
System's Responsibility:
Prepare the input data for processing, and convert the data between different representations (vertexes, primitives, fragments) throughout the pipeline
Rasterize things, and make sure the final pixels output agree with what the programmer has defined
Take advantage of the GPU to do these operations efficiently
kayvonf
@hofstee. What do you mean by: "make sure the final pixels agree with what the programmer has defined?" That sounds like "meet the OpenGL specification" to me.
hofstee
@kayvonf I need a better way to describe per-sample processing stage of the pipeline.
mrrobot
As an addition to MapReduce, parallelism is expressed by the programmer through the map and reduce functions. If most of the keys generated by the mappers map to the same reducer, then the program will be less parallel compared to a version which distributes keys across several reducers. The framework takes care of difficult issues that the programmer would like to avoid like fault tolerance - Failed map tasks and incomplete reduce tasks are re-run by the master, reduce tasks which have completed need not be re-run (output is available via HDFS).
fleventyfive
SQL:
Programmer's Responsibility: Define what data you want to manipulate/read/write in the form of a query. (for instance, select * from users where name='fleventyfive')
System's Responsibility: Decide how the data is to be manipulated/read/written by developing a query plan, optimizing the query, building executors, accessing indexes, updating database state, and finally accessing the data store.
ok
System Verilog:
Programmer's Responsibility: Describing the hardware of the digital circuit.
System's Responsibility: Deciding where/how this hardware is created on the FPGA
For MapReduce:
Programmer's responsibility:
Define mappers, reducers and the data structures that's been processed using the provided framework.
(Optinally)Define how to partition the data in order to distribute the data into multiple machines.
System's responsibility:
For OpenGL:
Programmer's responsibility:
1. Specify how camera, lighting, transformation, texture, alpha_blending and all sorts of graphics setting are set up.
2. Specify list of primitives to be drawn in the scene and their properties like positions, normals and texture coordinates.
System's responsibility:
1. From programmer's input, figure out all the math for setting camera, lightning and so on.
2. Communicate with GPU to take advantage of its Graphics-specific units and parallelism to make rasterization more efficient.
@Khryl. I like your breakdown, but in modern OpenGL (e.g., OpenGL ES), there is no concept of a camera, lighting, or materials. This might seem unintuitive, but it's true!
I'll take a stab at modern OpenGL:
Programmer's responsibility:
System's Responsibility:
@hofstee. What do you mean by: "make sure the final pixels agree with what the programmer has defined?" That sounds like "meet the OpenGL specification" to me.
@kayvonf I need a better way to describe per-sample processing stage of the pipeline.
As an addition to MapReduce, parallelism is expressed by the programmer through the map and reduce functions. If most of the keys generated by the mappers map to the same reducer, then the program will be less parallel compared to a version which distributes keys across several reducers. The framework takes care of difficult issues that the programmer would like to avoid like fault tolerance - Failed map tasks and incomplete reduce tasks are re-run by the master, reduce tasks which have completed need not be re-run (output is available via HDFS).
SQL:
Programmer's Responsibility: Define what data you want to manipulate/read/write in the form of a query. (for instance, select * from users where name='fleventyfive')
System's Responsibility: Decide how the data is to be manipulated/read/written by developing a query plan, optimizing the query, building executors, accessing indexes, updating database state, and finally accessing the data store.
System Verilog:
Programmer's Responsibility: Describing the hardware of the digital circuit.
System's Responsibility: Deciding where/how this hardware is created on the FPGA