Configuring Hive on MR3 to access secure (Kerberized) HDFS requires the user to update several files:

  1. kubernetes/env.sh
  2. kubernetes/conf/core-site.xml
  3. kubernetes/conf/yarn-site.xml
  4. kubernetes/conf/krb5.conf.

In kubernetes/env.sh, the user specifies how to renew HDFS tokens:

USER_PRINCIPAL=hive@RED
USER_KEYTAB=$KEYTAB_MOUNT_DIR/hive.service.keytab

TOKEN_RENEWAL_HDFS_ENABLED=true
  • USER_PRINCIPAL specifies the principal name to use when renewing HDFS and Hive tokens in DAGAppMaster and ContainerWorkers (for mr3.principal in mr3-site.xml).
  • USER_KEYTAB specifies the name of the keytab file which should be copied to the directory kubernetes/key by the user. Note that we may reuse the service keytab file for connecting to Metastore.
  • TOKEN_RENEWAL_HDFS_ENABLED should be set to true in order to automatically renew HDFS tokens.

In kubernetes/conf/core-site.xml, the user specifies the Kerberos principal name for the HDFS NameNode and the address of the Hadoop key management server (KMS):

<property>
  <name>fs.defaultFS</name>
  <value>file:///</value>
</property>

<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>

<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>hdfs/red0@RED</value>
</property>

<property>
  <name>dfs.encryption.key.provider.uri</name>
  <value>kms://http@red0:9292/kms</value>
</property>
  • It is important that fs.defaultFS should be set file:///, not an HDFS address like hdfs://red0:8020. This is because from the viewpoint of HiveServer2 running in a Kubernetes cluster, the default file system is the local file system. In fact, HiveServer2 is not even aware that it is reading from a remote HDFS.
  • If the configuration key dfs.namenode.kerberos.principal is not specified, Metastore may generate java.lang.IllegalArgumentException, as in:
    org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException DestHost:destPort blue0:8020 , LocalHost:localPort hivemr3-metastore-0.metastore.hivemr3.svc.cluster.local/10.44.0.1:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name)
    
  • The configuration key dfs.encryption.key.provider.uri is unnecessary if no encryption is used in HDFS.

In kubernetes/conf/yarn-site.xml, the user specifies the principal name of the Yarn ResourceManager:

<property>
  <name>yarn.resourcemanager.principal</name>
  <value>rm/red0@RED</value>
</property>

kubernetes/conf/krb5.conf contains the information for Kerberos configuration:

[libdefaults]
  dns_lookup_realm = false
  ticket_lifetime = 24h
  renew_lifetime = 7d
  forwardable = true
  rdns = false
  default_realm = RED
  default_ccache_name = /tmp/krb5cc_%{uid}

[realms]
  RED = {
    admin_server = red0
    kdc = red0
  }

Usually it suffices to reuse /etc/krb5.conf if the node is already set up to use Kerberos.

KrbApErrException: Message stream modified

Because of a bug in Kerberos, Hive on MR3 may fail to authenticate even with valid keytab files. In such a case, it usually prints an error message KrbApErrException: Message stream modified:

org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
...
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hive4/gold7@PL from keytab /opt/mr3-run/key/hive4.service.keytab javax.security.auth.login.LoginException: Message stream modified (41)
...
Caused by: sun.security.krb5.internal.KrbApErrException: Message stream modified (41)
	at sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:101) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308) ~[?:1.8.0_242]
	at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447) ~[?:1.8.0_242]
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780) ~[?:1.8.0_242]

The root cause of the bug is unknown and the user should find a fix specific to the base image in kubernetes/ranger/Dockerfile. With openjdk:8-jre-slim or CentOS 7.x (e.g., centos:7.6.1810) as the base image, the user should use a workaround by removing the setting for renew_lifetime in krb5.conf, as in:

[libdefaults]
  dns_lookup_realm = false
  ticket_lifetime = 24h
  forwardable = true
# renew_lifetime = 7d
  rdns = false
  default_realm = RED
  default_ccache_name = /tmp/krb5cc_%{uid}