Using Custom Configuration Variables |

When customizing the installation of Hive on MR3 on Kubernetes, the user may want to introduce additional configuration variables in kubernetes/env.sh so as to facilitate switching between different settings without having to update configuration files (hive-site.xml, tez-site.xml, mr3-site.xml, and core-site.xml).

As an example, suppose that the user often needs to change the configuration key fs.s3a.aws.credentials.provider in kubernetes/conf/core-site.xml, e.g.:

to com.amazonaws.auth.EnvironmentVariableCredentialsProvider on Minikube
to com.amazonaws.auth.InstanceProfileCredentialsProvider on Amazon AWS

`kubernetes/env.sh`

In order to simplify switching between the two settings, introduce a new environment variable in kubernetes/env.sh whose value will be substituted for fs.s3a.aws.credentials.provider later. In our example, we introduce a new environment variable MY_CRED_PROVIDER.

$ vi kubernetes/env.sh

MY_CRED_PROVIDER=com.amazonaws.auth.EnvironmentVariableCredentialsProvider

`kubernetes/hive/hive/hive-setup.sh`

Then define a corresponding system property in the function hive_setup_metastore_update_hadoop_opts in kubernetes/hive/hive/hive-setup.sh, and set it to the new environment variable. In our example, we define a new system property my.cred.provider.

$ vi kubernetes/hive/hive/hive-setup.sh

function hive_setup_metastore_update_hadoop_opts {
   export HADOOP_OPTS="$HADOOP_OPTS \
...
-Dsun.security.jgss.debug=true \
-Dmy.cred.provider=$MY_CRED_PROVIDER"

Updating configuration files

Now a reference to the system property ${my.cred.provider} in any configuration file automatically expands to the value assigned to MY_CRED_PROVIDER in kubernetes/env.sh. Finally set the configuration key fs.s3a.aws.credentials.provider in kubernetes/conf/core-site.xml to the new system property.

$ vi kubernetes/conf/core-site.xml

<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <value>${my.cred.provider}</value>
</property>

After rebuilding the Docker image, the user can easily switch to another setting by changing the value for the environment variable MY_CRED_PROVIDER in kubernetes/env.sh.