Running Hive on MR3 on Amazon EKS/Fargate is similar to running on Amazon EKS.
Hence the user can configure Metastore and HiveServer2 as explained in
Configuring Metastore and HiveServer2,
and run Metastore and HiveServer2 as explained in
Running Metastore and HiveServer2.
In order to enable autoscaling,
it suffices to set the configuration key mr3.enable.auto.scaling
to true in kubernetes/conf/mr3-site.xml
.
Below we show a few restrictions specific to EKS/Fargate.
Accessing S3 buckets
To access S3 buckets, use IAM roles for ServiceAccounts created by eksctl
as explained in Accessing S3 buckets.
Then we use WebIdentityTokenCredentialsProvider
(instead of InstanceProfileCredentialsProvider
)
for the configuration key fs.s3a.aws.credentials.provider
in kubernetes/conf/core-site.xml
.
No instance storage
ContainerWorker Pods running on Fargate cannot use instance storage. Hence we cannot use hostPath volumes and should use only emptyDir volumes for ContainerWorker Pods.
$ vi kubernetes/conf/mr3-site.xml
<property>
<name>mr3.k8s.pod.worker.emptydirs</name>
<value>/opt/mr3-run/work-local-dir</value>
</property>
<property>
<name>mr3.k8s.pod.worker.hostpaths</name>
<value></value>
</property>
No privileged init containers
Fargate does not support privileged init containers.
Hence we cannot use the configuration keys mr3.k8s.pod.worker.security.context.sysctls
and
mr3.k8s.pod.worker.init.container.command
in kubernetes/conf/mr3-site.xml
.