In order to make the best use of the capability provided by AWS Fargate, we have developed a few guidelines for running Hive on MR3 on Amazon EKS/Fargate.

This page is work in progress and may be revised/expanded in the future.

Prefer small ContainerWorker Pods

In the context of running Hive on MR3, the key limitation of Fargate is that no instance storage can be attached to ContainerWorker Pods, thus limiting their disk bandwidth. As we have to use a single emptyDir volume for writing intermediate data, the only way to increase the aggregate disk bandwidth is to create more ContainerWorker Pods for the same resources (in terms of CPU cores and memory). This strategy works under the assumption that the disk bandwidth for a ContainerWorker Pod on Fargate is irrelevant to its resources. (Typically a ContainerWorker Pod can use about 50MB/s to 100MB/s of disk bandwidth.)

Another reason to prefer small ContainerWorker Pods is to increase the aggregate disk space for writing intermediate data. This is because every ContainerWorker Pod on Fargate receives 20GB of container image layer storage regardless of its resources. Thus the aggregate disk space is directly proportional to the number of ContainerWorker Pods, not their total resources.

For scale-out, create many ContainerWorker Pods at once

When autoscaling is enabled, the configuration key mr3.auto.scale.out.num.increment.containers specifies the number of ContainerWorker Pods to create during a scale-out operation. As creating a ContainerWorker Pod can take up to 2 minutes, setting mr3.auto.scale.out.num.increment.containers to 1 or a small value prevents fast autoscaling and defeats the purpose of deploying ContainerWorker Pods on Fargate. As a workaround, the user can create many ContainerWorker Pods at once, thus effectively achieving fast autoscaling.

For scale-in, the configuration key mr3.auto.scale.in.num.decrement.hosts specifies the number of ContainerWorker Pods to delete during a scale-in operation (because each ContainerWorker Pod is counted as a host). Since scale-in operations complete almost instantly, the user should set mr3.auto.scale.in.num.decrement.hosts after taking into account the property of his or her workload. For example, setting it to a large value may make sense if queries are executed only sporadically.

Store the Docker image for ContainerWorker Pods on Amazon ECR

As we expect frequent creation of ContainerWorker Pods, it is a good practice to store the Docker image on Amazon Elastic Container Registry (ECR) (in the same AWS region) so as to reduce the time for creating ContainerWorker Pods. In such a case, update the environment variable DOCKER_HIVE_WORKER_IMG in kubernetes/env.sh.

$ vi kubernetes/env.sh

DOCKER_HIVE_WORKER_IMG=111111111111.dkr.ecr.ap-northeast-1.amazonaws.com/hive3worker