This page shows how to use Helm and a pre-built Docker image available at DockerHub in order to operate Hive on MR3 with Minikube. All components (Metastore, HiveServer2, MR3 DAGAppMaster) will be running inside Minikube. For Metastore, we will run a MySQL database as a Pod inside Minikube. By following the instruction, the user will learn:
- how to start Metastore using Helm
- how to use Helm to run Hive on MR3 with Minikube
- how to create Beeline connections and send queries to HiveServer2 running inside Minikube
This scenario has the following prerequisites:
- A running Minikube cluster should be available.
- The user should be able to execute: 1) command
kubectl
; 2) commandhelm
to use Helm. - A MySQL connector should be available.
This scenario should take less than 30 minutes to complete,
not including the time for downloading a pre-built Docker image.
This page has been tested with MR3 release 1.2 on CentOS 7.5 running Minikube v1.2.0 using user gla
.
Installation
Download an MR3 release containing the executable scripts.
$ git clone https://github.com/mr3project/mr3-run-k8s.git
$ cd mr3-run-k8s/kubernetes/
$ git reset --hard 3287a3dcda7cdd4875fcc2b5e345bc9089f000dc
Starting a MySQL database
For simplicity, we will run a MySQL database for Metastore as a Pod inside Minikube.
$ helm install --name mysql --namespace hivemr3 stable/mysql
NAME: mysql
LAST DEPLOYED: Tue Oct 27 22:07:26 2020
NAMESPACE: hivemr3
STATUS: DEPLOYED
...
NOTES:
MySQL can be accessed via port 3306 on the following DNS name from within your cluster:
mysql.hivemr3.svc.cluster.local
...
mysql.hivemr3.svc.cluster.local
is the address (FQDN) of the MySQL database.
Retrieve the root password as follows:
$ kubectl get secret --namespace hivemr3 mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo
vs6ZZ6XME9
Linking configuration files
We will reuse the configuration files in conf/
(and keys in key
if Kerberos is used for authentication).
Create symbolic links.
$ mkdir -p key
$ ln -s $(pwd)/conf/ helm/hive/conf
$ ln -s $(pwd)/key/ helm/hive/key
Now any change to the configuration files in conf/
is honored when running Hive on MR3.
Creating local directories
We need to create two new local directories:
- for a PersistentVolume to be shared by Pods;
- for a hostPath volume for ContainerWorker Pods.
Create a local directory for the PersistentVolume.
$ mkdir -p /home/gla/workdir
$ chmod 777 /home/gla/workdir
Hive on MR3 uses local disks for writing intermediate data. In the case of running on Kubernetes, we mount hostPath volumes to mount directories of the local machine. For our example, we create a local directory for the hostPath volume for ContainerWorker Pods.
$ mkdir -p /data1/gla/k8s
$ chmod 777 /data1/gla/k8s
Preparing a MySQL connector
The user should have a MySQL connector compatible with the MySQL database for Metastore. One can download the official JDBC driver for MySQL at https://dev.mysql.com/downloads/connector/j/.
Copy the MySQL connector to the directory /lib
under the local directory for the PersistentVolume.
$ mkdir -p /home/gla/workdir/lib
$ cp mysql-connector-java-8.0.12.jar /home/gla/workdir/lib/
$ chmod 777 /home/gla/workdir/lib/mysql-connector-java-8.0.12.jar
Configuring Pods
Create a new file helm/hive/values-minikube.yaml
which is a collection of values to override those in helm/hive/values.yaml
.
$ vi helm/hive/values-minikube.yaml
docker:
image: mr3project/hive3:1.2
imagePullPolicy: IfNotPresent
create:
metastore: true
metastore:
databaseHost: mysql.hivemr3.svc.cluster.local
warehouseDir: file:///opt/mr3-run/work-dir/warehouse
initSchema: true
mountLib: true
secureMode: false
resources:
requests:
cpu: 1
memory: 4Gi
limits:
cpu: 1
memory: 4Gi
heapSize: 4096
hive:
createSecret: false
externalIp: 12.34.56.78 # use your IP address
authentication: NONE
resources:
requests:
cpu: 1
memory: 8Gi
limits:
cpu: 1
memory: 8Gi
heapSize: 8192
workDir:
isNfs: false
volumeStr: "hostPath:\n path: /home/gla/workdir"
docker.image
is set to a pre-built Docker imagemr3project/hive3:1.0
available at DockerHub.docker.imagePullPolicy
is set toIfNotPresent
because we download the Docker image from DockerHub.create.metastore
is set to true because we will create a Metastore Pod.metastore.databaseHost
is set to the address (FQDN) of the MySQL database.metastore.initSchema
is set to true because it is the first time to run Metastore. For subsequent runs, the user should set it to false.metastore.mountLib
is set to true because we mount a MySQL connector inside the Metastore Pod.hive.externalIp
is set to the public IP address of the local machine.workDir.volumeStr
is set to the path to the local directory for the PersistentVolume.
Update conf/mr3-site.xml
.
$ vi conf/mr3-site.xml
<property>
<name>mr3.k8s.pod.image.pull.policy</name>
<value>IfNotPresent</value>
</property>
<property>
<name>mr3.k8s.pod.worker.hostpaths</name>
<value>/data1/gla/k8s/</value>
</property>
mr3.k8s.pod.image.pull.policy
is set toIfNotPresent
because we download the Docker image from DockerHub.mr3.k8s.pod.worker.hostpaths
is set to the path to the local directory for the hostPath volume.
Update conf/hive-site.xml
.
$ vi conf/hive-site.xml
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>vs6ZZ6XME9</value>
</property>
<property>
<name>hive.metastore.pre.event.listeners</name>
<value></value>
</property>
<property>
<name>metastore.pre.event.listeners</name>
<value></value>
</property>
<property>
<name>hive.security.authorization.manager</name>
<value>org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory</value>
</property>
javax.jdo.option.ConnectionPassword
is set to the password of the MySQL database.hive.metastore.pre.event.listeners
andmetastore.pre.event.listeners
are set to empty because we do not enable security on the Metastore side.
Update conf/core-site.xml
.
$ vi conf/core-site.xml
<property>
<name>hadoop.security.authentication</name>
<value>simple</value>
</property>
hadoop.security.authentication
is set tosimple
in order to disable Kerberos for authentication.
Starting Hive on MR3
Before running HiveServer2, the user should remove the label node-role.kubernetes.io/master
from minikube
node.
This is because Hive on MR3 does not count the resources of master nodes
when estimating the resources for ContainerWorker Pods.
Since minikube
node, the only node in a Minikube cluster, is a master node,
we should demote it to an ordinary node in order to secure resources for ContainerWorker Pods.
Thus, in order to be able to create ContainerWorker Pods in minikube
node,
the user should execute the following command:
$ kubectl label node minikube node-role.kubernetes.io/master-
Before running HiveServer2, the user should also make sure that no ConfigMaps and Services exist in the namespace hivemr3
.
For example, the user may see ConfigMaps and Services left over from a previous run.
$ kubectl get configmaps -n hivemr3
NAME DATA AGE
mr3conf-configmap-master 1 16m
mr3conf-configmap-worker 1 16m
$ kubectl get svc -n hivemr3
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service-master-1237-0 ClusterIP 10.105.238.21 <none> 80/TCP 11m
service-worker ClusterIP None <none> <none> 11m
In such a case, manually delete these ConfigMaps and Services.
$ kubectl delete configmap -n hivemr3 mr3conf-configmap-master mr3conf-configmap-worker
$ kubectl delete svc -n hivemr3 service-master-1237-0 service-worker
Install Helm chart for Hive on MR3 with values-minikube.yaml
.
We use hivemr3
for the namespace.
$ helm install --namespace hivemr3 helm/hive -f helm/hive/values-minikube.yaml
NAME: filled-aardvark
LAST DEPLOYED: Wed Oct 28 08:53:03 2020
NAMESPACE: hivemr3
STATUS: DEPLOYED
...
==> v1/ConfigMap
NAME DATA AGE
client-am-config 4 0s
env-configmap 1 0s
hivemr3-conf-configmap 18 0s
...
Check if all ConfigMaps are non-empty.
If the DATA
column for hivemr3-conf-configmap
is 0,
try to remove unnecessary files in the directory conf
or helm/hive/conf
.
The user can find four Pods running in the Minikube cluster:
- MySQL database; 2) Metastore; 3) HiveServer2; MR3 DAGAppMaster.
$ kubectl get pods -n hivemr3
NAME READY STATUS RESTARTS AGE
hivemr3-hiveserver2-r4pck 0/1 Running 0 57s
hivemr3-metastore-0 1/1 Running 0 57s
mr3master-3551-0-7mcns 1/1 Running 0 26s
mysql-8569cdf6fc-jn5mr 1/1 Running 0 10h
HiveServer2 Pod hivemr3-hiveserver2-r4pck
soon becomes ready after the readiness probe contacts it.
Running Beeline
Download a sample dataset and copy it to the directory for the PersistentVolume.
$ wget https://github.com/mr3project/mr3-release/releases/download/v1.0/pokemon.csv
$ cp pokemon.csv /home/gla/workdir
$ chmod 777 /home/gla/workdir/pokemon.csv
The user can verify that the sample dataset is accessible inside the HiveServer2 Pod.
$ kubectl exec -n hivemr3 -it hivemr3-hiveserver2-r4pck -- /bin/bash
root@hivemr3-hiveserver2-r4pck:/opt/mr3-run/hive# ls /opt/mr3-run/work-dir/
7342c023-93a7-4bda-932f-3cb40937c7df_resources c18c5e0e-bcf0-4cc9-be48-c06678db258a_resources lib pokemon.csv root
root@hivemr3-hiveserver2-r4pck:/opt/mr3-run/hive# exit
The user may use any client program (such as beeline
) to connect to HiveServer2 which is running at port 9852.
$ beeline -u "jdbc:hive2://12.34.56.78:9852/;;;" -n hive -p hive
Alternatively the user can run Beeline inside the Hiveserver2 Pod.
$ kubectl exec -n hivemr3 -it hivemr3-hiveserver2-r4pck -- /bin/bash
root@hivemr3-hiveserver2-r4pck:/opt/mr3-run/hive# export USER=root
root@hivemr3-hiveserver2-r4pck:/opt/mr3-run/hive# /opt/mr3-run/hive/run-beeline.sh
Execute queries.
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> show databases;
...
+----------------+
| database_name |
+----------------+
| default |
+----------------+
1 row selected (0.119 seconds)
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> use default;
...
No rows affected (0.031 seconds)
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> CREATE TABLE pokemon (Number Int,Name String,Type1 String,Type2 String,Total Int,HP Int,Attack Int,Defense Int,Sp_Atk Int,Sp_Def Int,Speed Int) row format delimited fields terminated BY ',' lines terminated BY '\n' tblproperties("skip.header.line.count"="1");
...
No rows affected (0.598 seconds)
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> load data local inpath '/opt/mr3-run/work-dir/pokemon.csv' INTO table pokemon;
...
No rows affected (0.575 seconds)
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> select avg(HP) from pokemon;
...
+---------------------+
| _c0 |
+---------------------+
| 144.84882280049567 |
+---------------------+
1 row selected (11.747 seconds)
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> create table pokemon1 as select *, IF(HP>160.0,'strong',IF(HP>140.0,'moderate','weak')) AS power_rate from pokemon;
...
No rows affected (2.044 seconds)
0: jdbc:hive2://hivemr3-hiveserver2-r4pck:985> select COUNT(name), power_rate from pokemon1 group by power_rate;
...
+------+-------------+
| _c0 | power_rate |
+------+-------------+
| 363 | strong |
| 336 | weak |
| 108 | moderate |
+------+-------------+
3 rows selected (2.19 seconds)
The user can see that ContainerWorker Pods have been created.
$ kubectl get pods -n hivemr3
NAME READY STATUS RESTARTS AGE
hivemr3-hiveserver2-r4pck 1/1 Running 0 6m7s
hivemr3-metastore-0 1/1 Running 0 6m7s
mr3master-3551-0-7mcns 1/1 Running 0 5m36s
mr3worker-87f8-1 1/1 Running 0 100s
mr3worker-87f8-2 1/1 Running 0 49s
mysql-8569cdf6fc-jn5mr 1/1 Running 0 10h
The user can find the warehouse directory /home/gla/workdir/warehouse/
.
$ ls /home/gla/workdir/warehouse
pokemon pokemon1
Terminating Hive on MR3
In order to terminate Hive on MR3, the user should first delete the DAGAppMaster Pod and then delete Helm chart, not the other way. This is because deleting Helm chart revokes the ServiceAccount object which DAGAppMaster uses to delete ContainerWorker Pods. Hence, if the user deletes Helm chart first, all remaining Pods should be deleted manually.
Delete ReplicationController for DAGAppMaster which in turn deletes all ContainerWorker Pods automatically.
$ kubectl get replicationcontroller -n hivemr3
NAME DESIRED CURRENT READY AGE
hivemr3-hiveserver2 1 1 1 9m46s
mr3master-3551-0 1 1 1 9m15s
$ kubectl -n hivemr3 delete replicationcontroller mr3master-3551-0
replicationcontroller "mr3master-3551-0" deleted
Delete Helm chart.
$ helm delete filled-aardvark
release "filled-aardvark" deleted
After a while, the user can see that only the MySQL database Pod remains.
$ kubectl get pods -n hivemr3
NAME READY STATUS RESTARTS AGE
mysql-8569cdf6fc-jn5mr 1/1 Running 0 10h
Then stop the MySQL database.
$ helm delete --purge mysql
As the last step, the user will find that the following objects belonging to the namespace hivemr3
are still alive:
- two ConfigMaps
mr3conf-configmap-master
andmr3conf-configmap-worker
- Service for DAGAppMaster, e.g.,
service-master-3551-0
- Service
service-worker
$ kubectl get configmaps -n hivemr3
NAME DATA AGE
mr3conf-configmap-master 1 11m
mr3conf-configmap-worker 1 10m
$ kubectl get svc -n hivemr3
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service-master-3551-0 ClusterIP 10.101.69.38 <none> 80/TCP 11m
service-worker ClusterIP None <none> <none> 11m
These ConfigMaps and Services are not deleted by the command helm delete
because
they are created not by Helm but by HiveServer2 and DAGAppMaster.
Hence the user should delete these ConfigMaps and Services manually.
$ kubectl delete configmap -n hivemr3 mr3conf-configmap-master mr3conf-configmap-worker
configmap "mr3conf-configmap-master" deleted
configmap "mr3conf-configmap-worker" deleted
$ kubectl delete svc -n hivemr3 service-master-3551-0 service-worker
service "service-master-3551-0" deleted
service "service-worker" deleted