Specifying a Task queue

When MR3 uses capacity scheduling (see Task Scheduling), the user can use the configuration key hive.mr3.dag.queue.name to specify a Task queue for each query. For example, by setting hive.mr3.dag.queue.name to high, all subsequent queries are sent to a Task queue high in MR3. If hive.mr3.dag.queue.name is not set, the default value of default is used (which is safe because MR3 always maintains a Task queue default).

0: jdbc:hive2://blue0:9852/> set hive.mr3.dag.queue.name=high;

To allow users to specify Task queues, the configuration key hive.security.authorization.sqlstd.confwhitelist.append should be set to include hive.mr3.dag.queue.name before starting HiveServer2, e.g.:

$ vi kubernetes/conf/hive-site.xml

<property>
  <name>hive.security.authorization.sqlstd.confwhitelist.append</name>
  <value>hive\.querylog\.location.*|hive\.mr3\.map\.task.*|hive\.mr3\.reduce\.task.*|hive\.mr3\.dag\.queue\.name</value>
</property>

Dynamically updating specifications for capacity scheduling

The user can update specifications for capacity scheduling without restarting HiveServer2 in two steps.

First delete the current ContainerGroup by setting hive.mr3.container.stop.cross.dag.reuse to true and submitting a query. After the query completes, the user can check that no ContainerWorkers are running.

0: jdbc:hive2://blue0:9852/> set hive.mr3.container.stop.cross.dag.reuse=true;
0: jdbc:hive2://blue0:9852/> select count(distinct foo) from bar;

Next set the configuration key hive.mr3.dag.queue.capacity.specs to new specifications for capacity scheduling and create a new ContainerGroup by submitting a query. In order to keep the new ContainerGroup alive, set the configuration key hive.mr3.container.stop.cross.dag.reuse to false.

0: jdbc:hive2://blue0:9852/> set hive.mr3.container.stop.cross.dag.reuse=false;
0: jdbc:hive2://blue0:9852/> set hive.mr3.dag.queue.capacity.specs=high=30,medium=30,default=30,background=0;
0: jdbc:hive2://blue0:9852/> select count(distinct foo) from fido;

The user can check the specifications of the new ContainerGroup in the log of DAGAppMaster.

2022-08-14T14:11:49,776  INFO [IPC Server handler 22 on 8080] TaskAttemptQueue$: DAGSchedulerCapacity: high=30/Simple(high),medium=30/Simple(medium),default=30/Simple(default),background=0/Simple(background)

To allow users to specify Task queues, the configuration key hive.security.authorization.sqlstd.confwhitelist.append should be set to include hive.mr3.container.stop.cross.dag.reuse and hive.mr3.dag.queue.capacity.specs before starting HiveServer2, e.g.:

$ vi kubernetes/conf/hive-site.xml

<property>
  <name>hive.security.authorization.sqlstd.confwhitelist.append</name>
  <value>hive\.querylog\.location.*|hive\.mr3\.map\.task.*|hive\.mr3\.reduce\.task.*|hive\.mr3\.dag\.queue\.name|hive\.mr3\.container\.stop\.cross\.dag\.reuse|hive\.mr3\.dag\.queue\.capacity\.specs</value>
</property>
Calendar Sep 27, 2023