MR3 Features

In this section, we describe the major features of MR3 as an execution engine. Most of these features are unique to MR3 and not found in its predecessor MapReduce/Tez. One can try these features by changing configuration parameters in mr3-site.xml.

Below we assume some familiarity with such notions as DAG, Vertex, Task, and TaskAttempt. For clarity, we refer to a client program of MR3 as MR3Client, the main coordinator process of MR3 as DAGAppMaster, and worker processes of MR3 as ContainerWorkers.

• MR3Client
• DAGAppMaster and ContainerWorker Modes
• MR3Client Inside/Outside Kubernetes
• ContainerGroup
• Concurrent DAGs
• DaemonTask
• DAG/Task Scheduling
• Worker Scheduling
• Fault Tolerance
• Speculative Execution
• Autoscaling
• MR3 Shuffle Handler
• Managing Fetchers
• Eliminating Fetch Delays
• Backpressure Handling
• Node Blacklisting
• Remote Shuffle Service