By running HiveServer2, we create a Kubernetes cluster depicted in the following diagram:

hive.k8s.hiveserver2

In order to run HiveServer2, the user can execute the script kubernetes/run-hive.sh:

$ kubernetes/run-hive.sh
...
CLIENT_TO_AM_TOKEN_KEY=668123ae-de9d-4ca3-95a6-a5848e123e6e
MR3_APPLICATION_ID_TIMESTAMP=10403
MR3_SHARED_SESSION_ID=f214e200-f38e-4b94-89d5-e0245de3dea5
ATS_SECRET_KEY=0d2fdeec-c564-4d40-891a-5ca5f736294c
configmap/client-am-config created
replicationcontroller/hivemr3-hiveserver2 created
service/hiveserver2 created

The script mounts the following files inside the HiveServer2 Pod:

  • kubernetes/env.sh
  • kubernetes/conf/*
  • kubernetes/key/*

In this way, the user can completely specify the behavior of HiveServer2 as well as DAGAppMaster and ContainerWorkers. For logging configuration, HiveServer2 reads kubernetes/conf/hive-log4j2.properties. DAGAppMaster and ContainerWorkers read k8s-mr3-container-log4j2.properties which is included in the MR3 release. By default, logging messages are redirected to console.

Executing the script kubernetes/run-hive.sh starts a HiveServer2 Pod and a DAGAppMaster Pod. It may take a while for the two Pods to become ready because both Pods run readiness and liveness probes. The HiveServer2 Pod becomes ready when it opens a Thrift port and starts accepting connection requests from Beeline. The DAGAppMaster Pod becomes ready when it opens an RPC port and starts accepting connection requests from MR3Client. The HiveServer2 Pod becomes ready only after the DAGAppMaster Pod becomes ready.

$ kubectl get pods -n hivemr3
NAME                        READY   STATUS    RESTARTS   AGE
hivemr3-hiveserver2-lmngh   1/1     Running   0          41s
mr3master-6196-0-dwnck      1/1     Running   0          30s

The user can verify that all files are successfully mounted inside the HiveServer2 Pod:

$ kubectl exec -n hivemr3 -it hivemr3-hiveserver2-lmngh -- /bin/bash
bash-4.2$ pwd
/opt/mr3-run/hive
bash-4.2$ cd /opt/mr3-run/
bash-4.2$ ls env.sh
env.sh
bash-4.2$ ls conf/
core-site.xml           hive-log4j2.properties.console  jgss.conf        mr3-site.xml              ranger-policymgr-ssl.xml
hive-log4j.properties   hive-log4j2.properties.file     krb5.conf        ranger-hive-audit.xml     tez-site.xml
hive-log4j2.properties  hive-site.xml                   mapred-site.xml  ranger-hive-security.xml  yarn-site.xml
bash-4.2$ ls key/
hive.service.keytab

The user can start a new Beeline connection using the address and service principal name of HiveServer2 (e.g., beeline -u "jdbc:hive2://10.1.91.41:9852/;principal=hive/red0@RED;"). After accepting queries from Beeline connections, DAGAppMaster creates many ContainerWorker Pods each of which runs a ContainerWorker container.

$ kubectl get pods -n hivemr3
NAME                        READY   STATUS     RESTARTS   AGE
hivemr3-hiveserver2-lmngh   1/1     Running    0          4m2s
mr3master-6196-0-dwnck      1/1     Running    0          3m51s
mr3worker-14e3-1            1/1     Running    0          17s
mr3worker-14e3-2            1/1     Running    0          11s
mr3worker-14e3-3            0/1     Init:0/1   0          5s
mr3worker-14e3-4            0/1     Init:0/1   0          5s

With hive.server2.enable.doAs set to true in hive-site.xml, the user should allow user root to impersonate potential clients by extending core-site.xml on the node where the Yarn ResourceManager is running, not kubernetes/conf/core-site.xml, where we assume that the user in the service principal name for HiveServer2 is root. For example, in order to accept queries from user foo, we could extend core-site.xml as follows:

hadoop.proxyuser.root.groups = foo
hadoop.proxyuser.root.hosts = red0

Here Metastore is running on node red0 (where impersonating user foo actually takes place).

While HiveServer2 is running, its log may repeatedly print ERROR messages due to org.apache.thrift.transport.TSaslTransportException.

2020-07-07T18:24:14,516 ERROR [HiveServer2-Handler-Pool: Thread-39] server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
...
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
...

This message is printed when the liveness probe checks the Thrift port, so it is not an error.

In order to stop HiveServer2, the user can delete ReplicationController for HiveServer2.

$ kubectl get -n hivemr3 replicationcontrollers
NAME                  DESIRED   CURRENT   READY   AGE
hivemr3-hiveserver2   1         1         1       10m
mr3master-6196-0      1         1         1       10m
$ kubectl -n hivemr3 delete replicationcontroller hivemr3-hiveserver2
replicationcontroller "hivemr3-hiveserver2" deleted

Deleting ReplicationController for HiveServer2 does not automatically terminate the DAGAppMaster Pod. This is a feature, not a bug, which is due to the support of high availability in Hive on MR3. (After setting environment variable MR3_APPLICATION_ID_TIMESTAMP properly, running run-hive.sh attaches the existing DAGAppMaster Pod to the new HiveServer2 Pod.)

Deleting DAGAppMaster Pod automatically deletes all ContainerWorker Pods as well, but another DAGAppMaster Pod is created shortly because we use ReplicationController for DAGAppMaster. In the following example, mr3master-6196-0-6qd4m is the second DAGAppMaster Pod which is created after deleting the initial DAGAppMaster Pod mr3master-6196-0-dwnck.

$ kubectl delete pod -n hivemr3 mr3master-6196-0-dwnck
pod "mr3master-6196-0-dwnck" deleted
$ kubectl get pods -n hivemr3
NAME                        READY   STATUS    RESTARTS   AGE
hivemr3-hiveserver2-lmngh   1/1     Running   0          9m10s
mr3master-6196-0-6qd4m      1/1     Running   0          47s

In order to stop DAGAppMaster, the user can delete ReplicationController for DAGAppMaster.

$ kubectl -n hivemr3 delete replicationcontroller mr3master-6196-0
replicationcontroller "mr3master-6196-0" deleted

After a while, no Pods should be running in the namespace hivemr3. To delete all remaining resources, execute the following command:

$ kubectl -n hivemr3 delete configmap --all; kubectl -n hivemr3 delete svc --all; kubectl -n hivemr3 delete secret --all; kubectl -n hivemr3 delete serviceaccount hive-service-account; kubectl -n hivemr3 delete role --all; kubectl -n hivemr3 delete rolebinding --all; kubectl delete clusterrole node-reader; kubectl delete clusterrolebinding hive-clusterrole-binding; kubectl -n hivemr3 delete persistentvolumeclaims workdir-pvc; kubectl delete persistentvolumes workdir-pv