Running Hive on MR3 on Amazon EKS/Fargate is similar to running on Amazon EKS. Hence the user can configure Metastore and HiveServer2 as explained in Configuring Metastore and HiveServer2, and run Metastore and HiveServer2 as explained in Running Metastore and HiveServer2. In order to enable autoscaling, it suffices to set the configuration key mr3.enable.auto.scaling to true in kubernetes/conf/mr3-site.xml. Below we show a few restrictions specific to EKS/Fargate.

Accessing S3 buckets

To access S3 buckets, use IAM roles for ServiceAccounts created by eksctl as explained in Accessing S3 buckets. Then we use WebIdentityTokenCredentialsProvider (instead of InstanceProfileCredentialsProvider) for the configuration key fs.s3a.aws.credentials.provider in kubernetes/conf/core-site.xml.

No instance storage

ContainerWorker Pods running on Fargate cannot use instance storage. Hence we cannot use hostPath volumes and should use only emptyDir volumes for ContainerWorker Pods.

$ vi kubernetes/conf/mr3-site.xml

<property>
  <name>mr3.k8s.pod.worker.emptydirs</name>
  <value>/opt/mr3-run/work-local-dir</value>
</property>

<property>
  <name>mr3.k8s.pod.worker.hostpaths</name>
  <value></value>
</property>

No privileged init containers

Fargate does not support privileged init containers. Hence we cannot use the configuration keys mr3.k8s.pod.worker.security.context.sysctls and mr3.k8s.pod.worker.init.container.command in kubernetes/conf/mr3-site.xml.

Calendar Sep 24, 2020