In Hive on MR3 on Kubernetes, a DAGAppMaster runs in one of the following three modes:

  • LocalThread mode: A DAGAppMaster starts as a new thread inside the MR3Client (typically HiveServer2).
  • LocalProcess mode: A DAGAppMaster starts as a new process on the same Pod where the MR3Client is running.
  • Kubernetes mode: A DAGAppMaster starts on its own Pod.

A ContainerWorker runs in one of the following two modes:

  • Local mode: A ContainerWorker starts as a thread inside the DAGAppMaster.
  • Kubernetes mode: A ContainerWorker starts as a Pod in a Kubernetes cluster.

Hive on MR3 support all combinations of these modes. Here are schematic descriptions of the six combinations. Note that in the current implementation, HiveServer2 always starts in a Pod.

  1. DAGAppMaster in LocalThread mode, ContainerWorker in Local mode masterworkermode1  

  2. DAGAppMaster in LocalThread mode, ContainerWorker in Kubernetes mode masterworkermode2  

  3. DAGAppMaster in LocalProcess mode, ContainerWorker in Local mode masterworkermode3  

  4. DAGAppMaster in LocalProcess mode, ContainerWorker in Kubernetes mode masterworkermode4

  5. DAGAppMaster in Kubernetes mode, ContainerWorker in Local mode masterworkermode5

  6. DAGAppMaster in Kubernetes mode, ContainerWorker in Kubernetes mode masterworkermode6

A DAGAppMaster mode can be specified with key mr3.master.mode in mr3-site.xml:

  • LocalThread mode: mr3.master.mode=local-thread
  • LocalProcess mode: mr3.master.mode=local-process
  • Kubernetes mode: mr3.master.mode=kubernetes

A ContainerWorker mode can be specified with key mr3.am.worker.mode in mr3-site.xml:

  • Local mode: mr3.am.worker.mode=local
  • Kubernetes mode: mr3.am.worker.mode=kubernetes
Calendar Oct 30, 2020