Before creating Pods for Hive on MR3, the user should configure Kubernetes objects such as namespace, Roles, RoleBindings, ServiceAccount, PersistentVolume, PersistentVolumeClaim, and so on. The following files specify how to configure Kubernetes objects common to all Pods:

└── kubernetes
    ├── env.sh
    └── yaml
        ├── cluster-role.yaml
        ├── hive-role.yaml
        ├── hiveserver2-service.yaml
        ├── hive-service-account.yaml
        ├── namespace.yaml
        ├── workdir-pvc.yaml
        └── workdir-pv.yaml

The user should also update the configuration files for Hive on MR3 itself. For Metastore, HiveServer2, DAGAppMaster, and ContainerWorker, these configuration files are found in the directory kubernetes/conf:

└── kubernetes
    └── conf
        ├── core-site.xml
        ├── hive-log4j2.properties
        ├── hive-log4j.properties
        ├── hive-site.xml
        ├── jgss.conf
        ├── krb5.conf
        ├── mapred-site.xml
        ├── mr3-site.xml
        ├── ranger-hive-audit.xml
        ├── ranger-hive-security.xml
        ├── ranger-policymgr-ssl.xml
        ├── tez-site.xml
        └── yarn-site.xml

For most of the configuration keys, the user can set their values in the same way as in Hive on MR3 on Hadoop. Those configuration keys specific to Hive on MR3 on Kubernetes are explained later.

Updating kubernetes/env.sh

The user should set the following environment variables in kubernetes/env.sh.

$ vi kubernetes/env.sh

MR3_NAMESPACE=hivemr3
MR3_SERVICE_ACCOUNT=hive-service-account
MASTER_SERVICE_ACCOUNT=master-service-account
WORKER_SERVICE_ACCOUNT=worker-service-account
CONF_DIR_CONFIGMAP=hivemr3-conf-configmap

CREATE_KEYTAB_SECRET=true   
KEYTAB_SECRET=hivemr3-keytab-secret
CREATE_WORKER_SECRET=true
WORKER_SECRET=hivemr3-worker-secret
  • MR3_NAMESPACE specifies the namespace for all Kubernetes objects.
  • MR3_SERVICE_ACCOUNT specifies the ServiceAccount for Hive on MR3.
  • MASTER_SERVICE_ACCOUNT specifies the ServiceAccount for MR3 DAGAppMaster.
  • WORKER_SERVICE_ACCOUNT specifies the ServiceAccount for MR3 ContainerWorkers.
  • CONF_DIR_CONFIGMAP specifies the name of the ConfigMap to be built from files in the directory kubernetes/conf.
  • CREATE_KEYTAB_SECRET specifies whether or not to create a Secret from files in the directory kubernetes/key. It should be set to true if Kerberos is used for authentication.
  • KEYTAB_SECRET specifies the name of the Secret to be built when CREATE_KEYTAB_SECRET is set to true.
  • CREATE_WORKER_SECRET specifies whether or not to create a Secret for MR3 ContainerWorkers.
  • WORKER_SECRET specifies the name of the Secret to be built when CREATE_WORKER_SECRET is set to true.

Then the user should manually update yaml files in accordance with kubernetes/env.sh as well as Kubernetes cluster settings. Since yaml files do not read environment variables, the user should manually update these files. Below we show how to update yaml files. In most case, the default values are all okay to use, except for those configurations specific to each Kubernetes cluster. We use boldface where it is mandatory to specify new values to override default values.

Using Kerberos

To use Kerberos, the user should update kubernetes/conf/krb5.conf which contains the information for Kerberos configuration:

$ vi kubernetes/conf/krb5.conf

[libdefaults]
  dns_lookup_realm = false
  ticket_lifetime = 24h
  forwardable = true
  rdns = false
  default_realm = RED
  default_ccache_name = /tmp/krb5cc_%{uid}

[realms]
  RED = {
    admin_server = red0
    kdc = red0
  }

Usually it suffices to reuse /etc/krb5.conf if the node is already set up to use Kerberos.

KrbApErrException: Message stream modified or KrbException: Message stream modified

Because of a bug in Kerberos, Hive on MR3 may fail to authenticate even with valid keytab files. In such a case, it usually prints an error message KrbApErrException: Message stream modified:

org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
...
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hive4/gold7@PL from keytab /opt/mr3-run/key/hive4.service.keytab javax.security.auth.login.LoginException: Message stream modified (41)
...
Caused by: sun.security.krb5.internal.KrbApErrException: Message stream modified (41)
	at sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:101) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447) ~[?:1.8.0_242]
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780) ~[?:1.8.0_242]

The root cause of the bug is unknown and the user should find a fix specific to the base image in kubernetes/ranger/Dockerfile. With openjdk:8-jre-slim or CentOS 7.x (e.g., centos:7.6.1810) as the base image, the user should use a workaround by removing the setting for renew_lifetime in krb5.conf, as in:

$ vi kubernetes/conf/krb5.conf

[libdefaults]
  dns_lookup_realm = false
  ticket_lifetime = 24h
  forwardable = true
# renew_lifetime = 7d
  rdns = false
  default_realm = RED
  default_ccache_name = /tmp/krb5cc_%{uid}

Updating YAML files

namespace.yaml

This file creates a namespace for all Kubernetes objects. The name field should match the namespace specified in MR3_NAMESPACE in kubernetes/env.sh.

$ vi kubernetes/yaml/namespace.yaml

  name: hivemr3

Similarly the namespace field in following yaml files should also match the same namespace: hive-service-account.yaml, hive-role.yaml, hiveserver2-service.yaml, workdir-pvc.yaml.

hive-service-account.yaml

This file creates a ServiceAccount. The name of the ServiceAccount object (hive-service-account) is read in run-hive.sh, so there is no need to update this file.

cluster-role.yaml

This file creates a ClusterRole. The name of the ClusterRole resource (node-reader) is read in run-hive.sh, so there is no need to update this file.

hive-role.yaml

This file creates a Role for HiveServer2 Pods. The name of the Role resource (hive-role) is read in run-hive.sh, so there is no need to update this file.

workdir-pv.yaml

This file creates a PersistentVolume for copying the result of running a query from ContainerWorkers to HiveServer2. The user should update it in order to use a desired type of PersistentVolume.

workdir-pvc.yaml

This file creates a PersistentVolumeClaim which references the PersistentVolume created by workdir-pv.yaml. The user should specify the size of the storage:

$ vi kubernetes/yaml/workdir-pvc.yaml

      storage: 10Gi

hiveserver2-service.yaml

This file creates a Service for exposing HiveServer2 to the outside of the Kubernetes cluster. The user should specify a public IP address with a valid host name and a port number (with name thrift) for HiveServer2 so that clients can connect to it from the outside of the Kubernetes cluster. Another port number with name http should be specified if HTTP transport is enabled (by setting the configuration key hive.server2.transport.mode to all or http in kubernetes/conf/hive-site.xml). The host name is necessary in order for Ranger to securely communicate with HiveServer2.

$ vi kubernetes/yaml/hiveserver2-service.yaml

  ports:
  - protocol: TCP
    port: 9852
    targetPort: 9852
    name: thrift
  - protocol: TCP
    port: 10001
    targetPort: 10001
    name: http
  externalIPs:
  - 10.1.91.41

In our example, we use 10.1.91.41:9852 as the full address of HiveServer2. The user should make sure that the IP address exists with a vaild host name and is not already taken.

Updating kubernetes/conf/mr3-site.xml

Those configuration keys relevant to Hive on MR3 on Kubernetes are explained in the Kubernetes section of Configuring MR3. Hive on MR3 automatically sets the following configuration keys, so their values in kubernetes/conf/mr3-site.xml are ignored:

  • mr3.k8s.namespace
  • mr3.k8s.pod.master.serviceaccount
  • mr3.k8s.pod.worker.serviceaccount
  • mr3.k8s.pod.master.image
  • mr3.k8s.pod.master.user
  • mr3.k8s.master.working.dir
  • mr3.k8s.master.persistentvolumeclaim.mounts
  • mr3.k8s.pod.worker.image
  • mr3.k8s.pod.worker.user
  • mr3.k8s.worker.working.dir
  • mr3.k8s.worker.persistentvolumeclaim.mounts
  • mr3.k8s.conf.dir.configmap
  • mr3.k8s.conf.dir.mount.dir
  • mr3.k8s.keytab.secret
  • mr3.k8s.worker.secret
  • mr3.k8s.mount.keytab.secret
  • mr3.k8s.mount.worker.secret
  • mr3.k8s.keytab.mount.dir
  • mr3.k8s.keytab.mount.file

For the following configuration keys, their default values in kubernetes/conf/mr3-site.xml must be used:

  • mr3.k8s.master.command (with a default value of /opt/mr3-run/hive/run-master.sh)
  • mr3.k8s.worker.command (with a default value of /opt/mr3-run/hive/run-worker.sh)

For the following configuration keys, their default values in kubernetes/conf/mr3-site.xml usually suffice, but the user may need to update their values according to the setting of the Kubernetes cluster:

  • mr3.k8s.api.server.url
  • mr3.k8s.client.config.file
  • mr3.k8s.service.account.use.token.ca.cert.path
  • mr3.k8s.service.account.token.path
  • mr3.k8s.service.account.token.ca.cert.path
  • mr3.k8s.nodes.polling.interval.ms
  • mr3.k8s.pods.polling.interval.ms
  • mr3.k8s.pod.creation.timeout.ms
  • mr3.k8s.pod.master.node.selector
  • mr3.k8s.master.pod.affinity.match.label. A value hivemr3_app=hiveserver2 means that the DAGAppMaster Pod is likely to be placed on the same node that hosts a Pod with label hivemr3_app=hiveserver2, namely the HiveServer2 Pod.
  • mr3.k8s.pod.worker.node.selector

For the following configuration keys, the user should check their values before starting Hive on MR3:

  • mr3.k8s.pod.image.pull.policy
  • mr3.k8s.pod.image.pull.secrets
  • mr3.k8s.host.aliases

The user can use the following configuration keys to specify tolerations for DAGAppMaster and ContainerWorker Pods:

  • mr3.k8s.pod.master.toleration.specs
  • mr3.k8s.pod.worker.toleration.specs

Their values are a comma-separated list of toleration specifications. The format of a toleration specification is [key]:[operator]:[value]:[effect]:[toleration in seconds] where [value] and :[toleration in seconds] are optional. Here are a few valid examples: hello:Equal:world:NoSchedule, hello:Exists::NoSchedule, hello:Equal:world:NoExecute:300, hello:Exists::NoExecute:300. Note that a wrong specification fails the creation of DAGAppMaster Pod or ContainerWorker Pods, so the user should check the validity of every toleration specification before running HiveServer2. For example, foo:Equal::NoSchedule is a wrong specification because [value] must be empty when [operator] is Exists. (Cf. foo:Equal::NoSchedule is okay.)

The following configuration keys determine emptyDir and hostPath volumes to be mounted inside DAGAppMaster and ContainerWorker Pods, and thus should be updated for each installation of Hive on MR3:

  • mr3.k8s.pod.master.emptydirs
  • mr3.k8s.pod.master.hostpaths
  • mr3.k8s.pod.worker.emptydirs
  • mr3.k8s.pod.worker.hostpaths

For both DAGAppMaster and ContainerWorker Pods, emptyDir and hostPath volumes become local directories where intermediate data is written, so at least one such volume is necessary. For the DAGAppMaster Pod, the following setting is usually okay because DAGAppMaster needs just a single local directory (unless it uses Local mode for ContainerWorkers):

  • mr3.k8s.pod.master.emptydirs = /opt/mr3-run/work-local-dir
  • mr3.k8s.pod.master.hostpaths is set to empty.

For ContainerWorker Pods, the set of available local directories matters for performance. If the same set of local disks are mounted on every node in the Kubernetes cluster, the user can set mr3.k8s.pod.master.hostpaths to the list of directories from local disks while leaving mr3.k8s.pod.worker.emptydirs to empty. For example, the following setting is appropriate for a homogeneous Kubernetes cluster in which three local disks are mounted on every node:

  • mr3.k8s.pod.worker.emptydirs is set to empty
  • mr3.k8s.pod.worker.hostpaths = /data1/k8s,/data2/k8s,/data3/k8s

Note that the user should never use more than one directory from each local disk because it only degrades the performance of writing to local disks. If no such local disks are attached, mr3.k8s.pod.worker.hostpaths should be set to empty and the user should use an emptyDir volume for writing intermediate data:

  • mr3.k8s.pod.worker.emptydirs = /opt/mr3-run/work-local-dir
  • mr3.k8s.pod.worker.hostpaths is set to empty.

For connecting to Metastore

The following code shows part of kubernetes/env.sh for specifying parameters for connecting to Metastore:

$ vi kubernetes/env.sh

HIVE_DATABASE_HOST=red0
HIVE_METASTORE_HOST=hivemr3-metastore-0.metastore.hivemr3.svc.cluster.local
HIVE_METASTORE_PORT=9850
HIVE_DATABASE_NAME=hive3mr3

HIVE_WAREHOUSE_DIR=/opt/mr3-run/work-dir/warehouse/

METASTORE_SECURE_MODE=true
HIVE_METASTORE_KERBEROS_PRINCIPAL=hive/indigo20@RED
HIVE_METASTORE_KERBEROS_KEYTAB=$KEYTAB_MOUNT_DIR/hive.service.keytab

  • HIVE_DATABASE_HOST specifies the host where the database server for Metastore is running.

  • HIVE_METASTORE_HOST and HIVE_METASTORE_PORT specify the address of Metastore itself.

    As we want to create a Metastore Pod, set HIVE_METASTORE_HOST to hivemr3-metastore-0.metastore.hivemr3.svc.cluster.local. Here hivemr3-metastore-0 is the unique name of the Pod that will be running Metastore, and hivemr3 is the namespace.

    To use an existing Metastore running as an external component (without creating a new Metastore Pod), set HIVE_METASTORE_HOST to its host (e.g., red0).

  • HIVE_WAREHOUSE_DIR specifies the path to the Hive warehouse. Since MR3 is agnostic to the type of data sources, it is important to specify the full path to the warehouse, including the file system. If no file system is given, MR3 assumes the local file system because the configuration key fs.defaultFS is set to file:/// in kubernetes/conf/core-site.xml.

    Below are a few examples of the path. For running Hive on MR3 in a Kubernetes cluster, the user should use either hdfs or s3a for the file system.

    • /opt/mr3-run/work-dir/warehouse/: a local directory inside the HiveServer2 Pod is used for the Hive warehouse. Since the local directory is not visible to the outside, this works only if all the components (HiveServer2, DAGAppMaster, and ContainerWorkers) run in the same Pod.
    • hdfs://red0:8020/user/hive/warehouse: an HDFS directory with NameNode on red0 is used for the Hive warehouse.
    • s3a://mr3-bucket/warehouse: an S3 bucket is used for the Hive warehouse.

  • If Metastore uses Kerberos-based authentication and runs in a secure mode, METASTORE_SECURE_MODE should be set to true. HIVE_METASTORE_KERBEROS_PRINCIPAL specifies the service principal name, and HIVE_METASTORE_KERBEROS_KEYTAB specifies the name of the service keytab file which should be copied to the directory kubernetes/key by the user. Note that if HiveServer2 uses Kerberos-based authentication, METASTORE_SECURE_MODE should also be set to true.

If HIVE_DATABASE_HOST and HIVE_METASTORE_HOST use hosts unknown to the default DNS, the user should add their aliases in the spec/template/spec/hostAliases of kubernetes/yaml/hive.yaml. The following example adds host names red0 and indigo20 that are unknown to the default DNS.

$ vi kubernetes/yaml/hive.yaml

spec:
  template:
    spec:
      hostAliases:
      - ip: "10.1.91.4"
        hostnames:
        - "red0"
      - ip: "10.1.91.41"
        hostnames:
        - "indigo20"

For connecting to HiveServer2

The following code shows part of kubernetes/env.sh for specifying parameters for connecting to HiveServer2:

$ vi kubernetes/env.sh

HIVE_SERVER2_HOST=$HOSTNAME
HIVE_SERVER2_PORT=9852
HIVE_SERVER2_HTTP_PORT=10001

HIVE_SERVER2_HEAPSIZE=32768

HIVE_SERVER2_AUTHENTICATION=KERBEROS
HIVE_SERVER2_KERBEROS_PRINCIPAL=hive/indigo20@RED
HIVE_SERVER2_KERBEROS_KEYTAB=$KEYTAB_MOUNT_DIR/hive.service.keytab

TOKEN_RENEWAL_HIVE_ENABLED=false
  • HIVE_SERVER2_PORT and HIVE_SERVER2_HTTP_PORT should match the port numbers specified in hiveserver2-service.yaml.
  • HIVE_SERVER2_HEAPSIZE specifies the heap size (in MB) for HiveServer2. If DAGAppMaster runs in LocalThread mode, the heap size should be no larger than the memory allocated to the Pod for running HiveServer2 (specified in hive.yaml). If DAGAppMaster runs in LocalProcess mode, the sum with the heap size of DAGAppMaster (specified by mr3.am.resource.memory.mb in conf/mr3-site.xml) should be no larger than the memory allocated to the Pod.
  • If HiveServer2 uses Kerberos-based authentication with HIVE_SERVER2_AUTHENTICATION set to KERBEROS, HIVE_SERVER2_KERBEROS_PRINCIPAL and HIVE_SERVER2_KERBEROS_KEYTAB should specify the service principal name and the service keytab file (for hive.server2.authentication.kerberos.principal and hive.server2.authentication.kerberos.keytab in hive-site.xml), respectively. Note that this service principal name may be different from the name in HIVE_METASTORE_KERBEROS_PRINCIPAL, and the service keytab file may be different from the file in HIVE_METASTORE_KERBEROS_KEYTAB.
  • TOKEN_RENEWAL_HIVE_ENABLED should be set to true in order to automatically renew Hive tokens.

Specifying ImagePullSecrets

By default, Hive on MR3 does not use ImagePullSecrets when downloading Docker images. The user can also use an existing ImagePullSecret in two steps.

First add a new field spec/template/spec/imagePullSecrets/name in hive.yaml:

$ vi kubernetes/yaml/hive.yaml

spec:
  template:
    spec:
      imagePullSecrets:
      - name: myregistrykey

Alternatively the user may add a new field imagePullSecrets in hive-service-account.yaml.

Then specify the same secret in the configuration key mr3.k8s.pod.image.pull.secrets in kubernetes/conf/mr3-site.xml:

$ vi kubernetes/conf/mr3-site.xml

<property>
  <name>mr3.k8s.pod.image.pull.secrets</name>
  <value>myregistrykey</value>
</property>

(Alternatively the user may add a new field imagePullSecrets in master-service-account.yaml and worker-service-account.yaml.)

Similarly the user should update ats.yaml, ranger.yaml, and metastore.yaml in order to use an existing ImagePullSecret.

Calendar Feb 21, 2023