Previous | Next --- Slide 7 of 42
Back to Lecture Thumbnails
ghawk

Extending the footnote comment on evaluation for another Programming System, I'd like to point out the Programmer's and System's responsibility for MapReduce:

Programmer's Responsibility

  • Describe the Map and Reduce tasks by extending the classes (along with the input and output type for the classes, where the mapper's output matches the reducer's input)
  • Describe the workflow of multiple MapReduce stages, if there exists a dependency of MapReduce jobs

System's Responsibility

  • Distributing the files across the cluster with redundancy (HDFS)
  • Schedule Map and Reduce tasks across nodes in the cluster (by load balancing effectively), marshaling data between the map and reduce tasks (and reducing this communication by localizing tasks based on the location of the data).
  • Shuffle keys of output pairs from Mapper to balance the distribution across reducers
  • Fault tolerance of Map and Reduce tasks
  • Deciding the file split sizes and no. of mappers/reducers (these parameters can be configured by the Programmer too)
iZac

OpenGL:

Programmer's Responsibility: Specify commands to build up the desired model from a small set of geometric primitives - points, lines, and polygons.

System's Responsibility: Provides portability- Produces consistent visual display results on any OpenGL API-compliant hardware, regardless of operating system or windowing system. Programmer does not need to design based on particular hardware features.

cube

SQL:

Programmer: Specify which data you would like to perform an operation on, and which operation should be performed on that data. Specify datatypes for each of the columns in the database, and how they should be indexed by the server.

System: Create and index the tables correctly. When given a query, run it in the most efficient manner and get a correct result back to the user. Utilize parallelism and provide portability. Programmer shouldn't have to worry about what system they're running on. SQL servers also provide authentication and user management for the servers themselves.

lament

Why has no one mentioned LISP? Back in the day, they used to sell computers dedicated to just running LISP, but today, kids don't even get jokes about having too many parentheses.

rokislt10

OpenGL:

Programmer: Setting vertices, creating triangles and meshes, basically building the world and setting the viewing angle

System: Schedules parallel computation, computes what color each pixel should be