Question

MapReduce and Hadoop

(a) Explain the difference between map and reduce tasks in the MapReduce framework.

(b) How does the Hadoop framework ensure that no reduce tasks can begin until all map tasks have finished?

(c) When a worker node fails in Hadoop, its tasks are reassigned to other workers. What guarantees that the data being processed by the failed node is available to these other workers?

Answer 1

a)

MapReduce is a framework using which we can write applications to process huge amounts of data, in parallel, on large clusters of commodity hardware in a reliable manner.

The MapReduce algorithm contains two important tasks, namely Map and Reduce. Map takes a set of data and converts it into another set of data, where individual elements are broken down into tuples (key/value pairs). Secondly, reduce task, which takes the output from a map as an input and combines those data tuples into a smaller set of tuples. As the sequence of the name MapReduce implies, the reduce task is always performed after the map job.

b)

1. A Task Tracker is a slave node which accepts and executes the tasks from Job Tracker. Task Tracker runs in its own JVM Process. Every Task Tracker is configured with a set number of slots which indicates the number of tasks it can accept. The Task Tracker starts a separate JVM processes to do the actual work (called as Task Instance), this is to ensure that process failure does not take down the Task Tracker. The Task Tracker monitors these task instances, capturing the output and exit codes. When the Task instances finish, successfully or not, the task tracker notifies the Job Tracker. The Task Trackers also send out heartbeat messages to the Job Tracker, usually every few minutes, to reassure the Job Tracker that it is still alive. These messages also update the Job Tracker about the number of slots available.

c)

On a system of this scale, failure is common place. It is the job of the Master to periodically ping the Workers. If a Worker doesn't answer, it is marked as bad, and its work is rescheduled to another Worker. Furthermore, any Reduce that was scheduled to get results from the old Worker is told to begin getting them from the new Worker, instead.

When a Map worker dies, it needs to be re-executed from scratch. The reason for this is the results are stored on the Worker's local disk and are now inaccessible to Reduces. But, should a Reduce Worker fail, its results remain available in the global file system.

Why the difference? Well, remember, the results of a Reduce are designed for consumption by the end user. Because of this, they are placed in a distributed file system such that the program can get to all of them in one place.

By contrast the results of the Map Workers are intended only for consumption by a particular worker, so they are left in place. Instead, the upsteam reduce Worker is told of their location by the Master and they suck in the data explicitly, in a location-aware way, by an RPC-like mechanism.