This page shows how to operate Hive on MR3 on Amazon EKS with autoscaling. By following the instruction, the user will learn:

1. how to create and configure an EKS cluster for running Hive on MR3 with autoscaling
2. how to run Hive on MR3 in the EKS cluster

This scenario has the following prerequisites:

1. The user can create an EKS cluster with the command eksctl.
2. The user can create and update IAM policies.
3. The user either can create EFS or has access to an S3 bucket for storing transient data.
4. The user can configure LoadBalancers.
5. A MySQL server for Metastore is running and accessible from the EKS cluster.
6. The MySQL server for Metastore is already populated with Hive databases.
7. The user has access to an S3 bucket storing the warehouse and all S3 buckets containing datasets.
8. The user can download a MySQL connector.
9. The user can run Beeline to connect to HiveServer2 running at a particular address.

Although this page is self-contained, we recommend the user to read the page on Autoscaling and the guide on running Hive on MR3 on Amazon EKS before proceeding. Since Amazon EKS is a particular case of Kubernetes, it also helps to try running Hive on MR3 on Minikube as explained in the page On Minikube with a Pre-built Docker Image.

We use a pre-built Docker image mr3project/hivemr3:1.2 available at DockerHub, so the user does not have to build a new Docker image. This scenario should take 1 hour to 2 hours to complete, including the time for creating an EKS cluster. This page has been tested with MR3 release 1.2 and eksctl 0.28.1.

Installation

Download an MR3 release containing the executable scripts.

$ git clone https://github.com/mr3project/mr3-run-k8s.git
$ cd mr3-run-k8s/kubernetes/
$ git reset --hard 3287a3dcda7cdd4875fcc2b5e345bc9089f000dc

Restore those files created for Amazon EKS.

$ mv -f env.sh.eks env.sh
$ mv -f conf/core-site.xml.eks conf/core-site.xml
$ mv -f conf/mr3-site.xml.eks conf/mr3-site.xml
$ mv -f conf/hive-site.xml.eks conf/hive-site.xml
$ mv -f yaml/metastore.yaml.eks yaml/metastore.yaml
$ mv -f yaml/hive.yaml.eks yaml/hive.yaml
$ mv -f yaml/ranger.yaml.eks yaml/ranger.yaml
$ mv -f yaml/ats.yaml.eks yaml/ats.yaml

IAM policy for autoscaling

Create an IAM policy EKSAutoScalingWorkerPolicy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Configuring the EKS cluster (eks/cluster.yaml)

Open eks/cluster.yaml. Set the region for an EKS cluster.

$ vi eks/cluster.yaml

metadata:
  region: ap-northeast-1

We create an EKS cluster with two node groups: mr3-master and mr3-worker. The mr3-master node group is intended for HiveServer2, DAGAppMaster, and Metastore Pods, and uses a single on-demand instance of type m5.xlarge for the master node. The mr3-worker node group is intended for ContainerWorker Pods, and uses up to three spot instances of type m5d.xlarge or m5d.2xlarge for worker nodes. Note that worker nodes have instance storage.

nodeGroups:
  - name: mr3-master
    instanceType: m5.xlarge
    desiredCapacity: 1
  - name: mr3-worker
    desiredCapacity: 0
    minSize: 0
    maxSize: 3
    instancesDistribution:
      instanceTypes: ["m5d.xlarge", "m5d.2xlarge"]
      onDemandBaseCapacity: 0
      onDemandPercentageAboveBaseCapacity: 0
      spotInstancePools: 1

The following diagram shows an example of the EKS cluster after launch:

In the iam/attachPolicyARNs field of both node groups mr3-master and mr3-worker, use the ARN (Amazon Resource Name) of the IAM policy created in the previous step. (Without using the ARN for mr3-master, the user cannot check the status of the Kubernetes Autoscaler.)

nodeGroups:
  - name: mr3-master
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::111111111111:policy/EKSAutoScalingWorkerPolicy
  - name: mr3-worker
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::111111111111:policy/EKSAutoScalingWorkerPolicy

In the preBootstrapCommands field of the node group mr3-master, replace your.server.address with the address for downloading a MySQL connector jar file which is necessary for connecting to the MySQL server for Metastore. In our example, we use mysql-connector-java-8.0.12.jar.

nodeGroups:
  - name: mr3-master
    preBootstrapCommands:
      - "wget http://your.server.address/mysql-connector-java-8.0.12.jar"
      - "mkdir -p /home/ec2-user/lib"
      - "mv mysql-connector-java-8.0.12.jar /home/ec2-user/lib"

In the preBootstrapCommands field of the node group mr3-worker, list commands for initializing instance storage. In our example, we format /dev/nvme1n1 as ext4 and mount a single local disk on the directory /ephemeral1.

nodeGroups:
  - name: mr3-worker
    preBootstrapCommands:
      - "mkfs -t ext4 /dev/nvme1n1"
      - "mkdir -p /ephemeral1"
      - "mount /dev/nvme1n1 /ephemeral1"
      - "chown ec2-user:ec2-user /ephemeral1"

Creating the EKS cluster

Create an EKS cluster by executing the command eksctl. By default, the command eksctl uses all Availability Zones (AZs) from the region specified by the field metadata/region in eks/cluster.yaml. The user should be aware that Amazon charges for data transfer between different AZs ($0.01/GB to $0.02/GB) and intermediate data exchanged by ContainerWorkers can cross the AZ boundary. The user can use a single AZ by updating eks/cluster.yaml as follows:

$ vi eks/cluster.yaml

availabilityZones: ["ap-northeast-1a", "ap-northeast-1c", "ap-northeast-1d"]

nodeGroups:
  - name: mr3-master
    availabilityZones: ["ap-northeast-1a"]
  - name: mr3-worker
    availabilityZones: ["ap-northeast-1a"]

If eksctl does not accepts this update, upgrade it to the latest version.

Creating an EKS cluster can take 15 minutes or longer.

$ eksctl create cluster -f eks/cluster.yaml
[ℹ]  eksctl version 0.28.1
[ℹ]  using region ap-northeast-1
[ℹ]  subnets for ap-northeast-1a - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for ap-northeast-1c - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for ap-northeast-1d - public:192.168.64.0/19 private:192.168.160.0/19
...
[✔]  EKS cluster "hive-mr3" in "ap-northeast-1" region is ready

The user can find that two Auto Scaling groups are created.

In our example, the mr3-master node group starts with a single master node whereas the mr3-worker node group starts with no node and can attach up to three nodes.

The user can verify that only the master node is available in the EKS cluster.

$ kubectl get nodes
NAME                                                STATUS   ROLES    AGE   VERSION
ip-192-168-18-120.ap-northeast-1.compute.internal   Ready    <none>   47s   v1.14.9-eks-cc7316

The user can get the public IP address of the master node.

$ kubectl describe node ip-192-168-18-120.ap-northeast-1.compute.internal
...
Addresses:
  InternalIP:   192.168.18.120
  ExternalIP:   18.183.130.42
  Hostname:     ip-192-168-18-120.ap-northeast-1.compute.internal
  InternalDNS:  ip-192-168-18-120.ap-northeast-1.compute.internal
  ExternalDNS:  ec2-18-183-130-42.ap-northeast-1.compute.amazonaws.com

Configuring Kubernetes Autoscaler (eks/cluster-autoscaler-autodiscover.yaml)

Open eks/cluster-autoscaler-autodiscover.yaml and change the configuration for autoscaling if necessary. By default, the Kubernetes Autoscaler removes nodes that stay idle for 1 minute (as specified by --scale-down-unneeded-time).

$ vi eks/cluster-autoscaler-autodiscover.yaml

spec:
  template:
    spec:
      containers:
          command:
            - --scale-down-delay-after-add=5m
            - --scale-down-unneeded-time=1m

Start the Kubernetes Autoscaler.

$ kubectl apply -f eks/cluster-autoscaler-autodiscover.yaml
serviceaccount/cluster-autoscaler created
clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created
role.rbac.authorization.k8s.io/cluster-autoscaler created
clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
deployment.apps/cluster-autoscaler created

The user can check that the Kubernetes Autoscaler has started properly.

$ kubectl logs -f deployment/cluster-autoscaler -n kube-system
...
I1029 06:33:48.128601       1 utils.go:543] Skipping ip-192-168-18-120.ap-northeast-1.compute.internal - no node group config
I1029 06:33:48.128645       1 static_autoscaler.go:393] Scale down status: unneededOnly=true lastScaleUpTime=2020-10-29 06:33:28.110776799 +0000 UTC m=+2.814747604 lastScaleDownDeleteTime=2020-10-29 06:33:28.110776872 +0000 UTC m=+2.814747680 lastScaleDownFailTime=2020-10-29 06:33:28.110776944 +0000 UTC m=+2.814747749 scaleDownForbidden=false isDeleteInProgress=false

Directory for storing transient data

The user can use either EFS or S3 to store transient data. For using EFS, follow the instruction in 1. Creating and mounting EFS below. For using S3, follow the instruction in 2. Using S3 instead of EFS below.

1. Creating and mounting EFS (efs/manifest.yaml)

The user can create EFS on the AWS Console. When creating EFS, choose the VPC of the EKS cluster. Make sure that a mount target is created for each Availability Zone. If the user can choose the security group for mount targets, use the security group for the EKS cluster (e.g., eks-cluster-sg-hive-mr3-1259748527). Otherwise identify two security groups:

1. the security group for the EC2 instances constituting the EKS cluster;
2. the security group associated with the EFS mount targets. If the two security groups are different, add a rule to allow inbound access using NFS from the first security group so that the EKS cluster can access EFS, as shown below: Here sg-096f3c3dff95ad6ae is the first security group.

Get the file system ID of EFS (e.g., fs-027c9d22) and the address (e.g., fs-027c9d22.efs.ap-northeast-1.amazonaws.com). Open efs/manifest.yaml and update the following fields.

$ vi efs/manifest.yaml

data:
  file.system.id: fs-027c9d22
  aws.region: ap-northeast-1
spec:
  template:
    spec:
      volumes:
      - name: pv-volume
        nfs:
          server: fs-027c9d22.efs.ap-northeast-1.amazonaws.com

If EKS does not accept the use of Deployment objects with apiVersion extensions/v1beta1, update efs/manifest.yaml as follows:

kind: Deployment
apiVersion: apps/v1
spec:
  selector:
    matchLabels:
      app: efs-provisioner

Execute the script mount-efs.sh to create a PersistentVolume.

$ ./mount-efs.sh 
namespace/hivemr3 created
serviceaccount/efs-provisioner created
clusterrole.rbac.authorization.k8s.io/efs-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-efs-provisioner created
role.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-efs-provisioner created
configmap/efs-provisioner created
deployment.extensions/efs-provisioner created
storageclass.storage.k8s.io/aws-efs created
persistentvolumeclaim/workdir-pvc created

The user can find a new StorageClass aws-efs, a new Pod, and a new PersistentVolumeClaim (workdir-pvc).

$ kubectl get sc
NAME            PROVISIONER             AGE
aws-efs         example.com/aws-efs     59s
gp2 (default)   kubernetes.io/aws-ebs   36m

$ kubectl get pods -n hivemr3
NAME                               READY   STATUS    RESTARTS   AGE
efs-provisioner-649ff654ff-msjzb   1/1     Running   0          25s

$ kubectl get pvc -n hivemr3
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
workdir-pvc   Bound    pvc-11c53ae3-19b5-11eb-ad3e-0ead4a1b8916   1Mi        RWX            aws-efs        39s

2. Using S3 instead of EFS

Set the configuration key hive.exec.scratchdir in conf/hive-site.xml to point to the directory on S3 for storing transient data. Note that we should use prefix s3a, not s3.

$ vi conf/hive-site.xml

<property>
  <name>hive.exec.scratchdir</name>
  <value>s3a://hivemr3-warehouse-dir/workdir/${user.name}</value>
</property>

If necessary, include an additional inline policy in the IAM roles for the mr3-master and mr3-worker node groups so that every Pod can access the S3 bucket for the scratch directory.

Open env.sh and set the following two environment variables to empty values.

$ vi env.sh

WORK_DIR_PERSISTENT_VOLUME_CLAIM=
WORK_DIR_PERSISTENT_VOLUME_CLAIM_MOUNT_DIR=

Open yaml/metastore.yaml and comment out the following lines:

$ vi yaml/metastore.yaml

# - name: work-dir-volume
#   mountPath: /opt/mr3-run/work-dir/
# - name: work-dir-volume
#   persistentVolumeClaim:
#     claimName: workdir-pvc

Open yaml/hive.yaml and comment out the following lines:

$ vi yaml/hive.yaml

# - name: work-dir-volume
#   mountPath: /opt/mr3-run/work-dir
# - name: work-dir-volume
#   persistentVolumeClaim:
#     claimName: workdir-pvc

Access to the MySQL server for Metastore

Check if the MySQL server for Metastore is accessible from the master node. If the MySQL server is running on Amazon AWS, the user may have to update its security group or VPC configuration.

Configuring Metastore

Open env.sh and set the environment variable HIVE_DATABASE_HOST to the address of the MySQL server for Metastore. In our example, the MySQL server is running at the IP address 3.112.194.145. Set the environment variable HIVE_WAREHOUSE_DIR to the S3 bucket storing the warehouse. Note that we should use prefix s3a, not s3.

$ vi env.sh

HIVE_DATABASE_HOST=3.112.194.145

HIVE_WAREHOUSE_DIR=s3a://hivemr3-warehouse-dir/warehouse

If the user sets the environment variable HIVE_DATABASE_HOST to the host name of the MySQL server, the user should update: 1) the spec/template/spec/hostAliases field in yaml/metastore.yaml and yaml/hive.yaml, and 2) the configuration key mr3.k8s.host.aliases in conf/mr3-site.xml.

Checking configurations for MR3

For conf/core-site.xml and conf/mr3-site.xml, the user should use the default values for all configuration keys. There are three configuration keys that the user may want to update.

• mr3.k8s.pod.worker.hostpaths in conf/mr3-site.xml is set to /ephemeral1 because the instance types m5d.xlarge and m5d.2xlarge for worker nodes are equipped with a single local disk mounted on the directory /ephemeral1. If the user uses different instance types with multiple local disks, the preBootstrapCommands field of the node group mr3-worker should be expanded to mount all local disks and the configuration key mr3.k8s.pod.worker.hostpaths should include additional directories.

• Since the Kubernetes Autoscaler is configured to remove nodes that remain idle for 1 minute for fast scale-in, mr3.auto.scale.in.grace.period.secs in conf/mr3-site.xml is set to 90 seconds (60 seconds of idle time and extra 30 seconds to account for delays). If the user wants to increase the value of --scale-down-unneeded-time in eks/cluster-autoscaler-autodiscover.yaml, the configuration key mr3.auto.scale.in.grace.period.secs should be adjusted accordingly.

• mr3.auto.scale.out.grace.period.secs in conf/mr3-site.xml should be set to a sufficiently large value so as to prevent MR3 from prematurely cancelling the provisioning of worker nodes. For example, if it takes about 3 minutes to create and initialize a new worker node, mr3.auto.scale.out.grace.period.secs can be set to 360 (equivalent to 6 minutes). For more details, see Autoscaling.

Configuring Metastore

Open conf/hive-site.xml and update configurations for Metastore as necessary. Below we list some of configuration keys that the user should check. The two configuration keys javax.jdo.option.ConnectionUserName and javax.jdo.option.ConnectionPassword should match the user name and password of the MySQL server for Metastore.

$ vi conf/hive-site.xml

<property>
  <name>hive.metastore.db.type</name>
  <value>MYSQL</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.cj.jdbc.Driver</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>passwd</value>
</property>

<property>
  <name>hive.security.metastore.authenticator.manager</name>
  <value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value>
</property>

<property>
  <name>hive.security.metastore.authorization.manager</name>
  <value>org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider</value>
</property>

<property>
  <name>hive.security.authenticator.manager</name>
  <value>org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator</value>
</property>

By default, Metastore does not initialize schema. In order to initialize schema when starting Metastore, update yaml/metastore.yaml as follows:

$ vi yaml/metastore.yaml

spec:
  template:
    spec:
      containers:
        args: ["start", "--kubernetes", "--init-schema"]

Docker image

By default, we use a pre-built Docker image mr3project/hivemr3:1.2. If the user wants to use a different Docker image, check the environment variable DOCKER_HIVE_IMG in env.sh and the field spec/template/spec/containers/image in yaml/metastore.yaml and yaml/hive.yaml.

$ vi env.sh

DOCKER_HIVE_IMG=${DOCKER_HIVE_IMG:-mr3project/hive3:1.2}

$ vi yaml/metastore.yaml

spec:
  template:
    spec:
      containers:
      - image: mr3project/hive3:1.2

$ vi yaml/hive.yaml

spec:
  template:
    spec:
      containers:
      - image: mr3project/hive3:1.2

Accessing S3 buckets

Find the IAM (Identity and Access Management) roles for the mr3-master and mr3-worker node groups (which typically look like eksctl-hive-mr3-nodegroup-mr3-mas-NodeInstanceRole-448MRIYIQ3F8 and eksctl-hive-mr3-nodegroup-mr3-wor-NodeInstanceRole-E19NHT8X0UJ7). For both IAM roles, add the following inline policy or its variant so that every Pod can access S3 buckets storing the warehouse and containing datasets. Adjust the Action field to restrict the set of operations permitted to Pods.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::hivemr3-warehouse-dir",
                "arn:aws:s3:::hivemr3-warehouse-dir/*",
                "arn:aws:s3:::hivemr3-partitioned-10-orc",
                "arn:aws:s3:::hivemr3-partitioned-10-orc/*"
            ]
        }
    ]
}

Running Metastore and HiveServer2

Depending on the characteristics of datasets and queries, the user may have to change configurations for Hive on MR3 before running Metastore and HiveServer2. Here are a couple of examples.

• Change resources to be allocated to each mapper, reducer, and ContainerWorker by updating conf/hive-site.xml. In particular, the configuration keys hive.mr3.all-in-one.containergroup.memory.mb and hive.mr3.all-in-one.containergroup.vcores should be adjusted so that a ContainerWorker can fit in a worker node. For example, we can use the following values for worker nodes of instance type m5d.xlarge.

  $ vi conf/hive-site.xml
  
  <property>
    <name>hive.mr3.all-in-one.containergroup.memory.mb</name>
    <value>15360</value>
  </property>
      
  <property>
    <name>hive.mr3.all-in-one.containergroup.vcores</name>
    <value>3</value>
  </property>
  

  For more details, see Performance Tuning.

• Change the parameters for accessing S3 to avoid SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool. For more details, see Troubleshooting.

The user can start Metastore by executing the script run-metastore.sh. After a Metastore Pod is created, the user can start HiveServer2 by executing the script run-hive.sh.

$ ./run-metastore.sh
Error from server (AlreadyExists): namespaces "hivemr3" already exists
...
configmap/client-am-config created
statefulset.apps/hivemr3-metastore created
service/metastore created

$ ./run-hive.sh 
...
replicationcontroller/hivemr3-hiveserver2 created
service/hiveserver2 created

HiveServer2 creates a DAGAppMaster Pod. Note, however, that no ContainerWorkers Pods are created until queries are submitted.

$ kubectl get pods -n hivemr3
NAME                               READY   STATUS    RESTARTS   AGE
efs-provisioner-649ff654ff-msjzb   1/1     Running   0          20m
hivemr3-hiveserver2-wcr5g          1/1     Running   0          89s
hivemr3-metastore-0                1/1     Running   0          102s
mr3master-8561-0-v9fmp             1/1     Running   0          73s

The user can check the log of the DAGAppMaster Pod to make sure that it has started properly.

$ kubectl logs -f -n hivemr3 mr3master-8561-0-v9fmp
...
2020-10-29T08:51:21,059  INFO [DAGAppMaster-1-13] HeartbeatHandler$: Timeout check in HeartbeatHandler:Container
2020-10-29T08:51:21,182  INFO [K8sContainerLauncher-3-1] K8sContainerLauncher: Resynchronizing Pod states for appattempt_8561_0000_000000: 0

Configuring the LoadBalancer

Executing the script run-hive.sh creates a new LoadBalancer for HiveServer2. Get the security group associated with the LoadBalancer. If necessary, edit the inbound rule in order to restrict the source IP addresses (by changing the source from 0.0.0.0/0 to (IP address)/32.

The LoadBalancer disconnects Beeline showing no activity for the idle timeout period, which is 60 seconds by default. The user may want to increase the idle timeout period, e.g., to 1200 seconds. Otherwise Beeline loses the connection to HiveServer2 even after a brief period of inactivity.

Running Beeline

To run Beeline, get the LoadBalancer Ingress of the Service hiveserver2.

$ kubectl describe service -n hivemr3 hiveserver2
Name:                     hiveserver2
Namespace:                hivemr3
Labels:                   <none>
Annotations:              <none>
Selector:                 hivemr3_app=hiveserver2
Type:                     LoadBalancer
IP:                       10.100.134.69
External IPs:             10.1.91.41
LoadBalancer Ingress:     aac514ccf19b911ebb0e306616e38843-1372236816.ap-northeast-1.elb.amazonaws.com
...

Get the IP address of the LoadBalancer Ingress.

$ nslookup aac514ccf19b911ebb0e306616e38843-1372236816.ap-northeast-1.elb.amazonaws.com
...
Non-authoritative answer:
Name:	aac514ccf19b911ebb0e306616e38843-1372236816.ap-northeast-1.elb.amazonaws.com
Address: 54.168.141.122
Name:	aac514ccf19b911ebb0e306616e38843-1372236816.ap-northeast-1.elb.amazonaws.com
Address: 13.115.227.34

In this example, the user can use 54.168.141.122 or 13.115.227.34 as the IP address of HiveServer2 when running Beeline. This is because Beeline connects first to the LoadBalancer, not directly to HiveServer2.

Here is an example of running Beeline using the address of 13.115.227.34 on another machine where the script hive/run-beeline.sh is installed.

$ hive/run-beeline.sh --tpcds --hivesrc3
...
Connecting to jdbc:hive2://13.115.227.34:9852/;;
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.2 by Apache Hive
0: jdbc:hive2://13.115.227.34:9852/>

After running a few queries, new worker nodes are attached and ContainerWorker Pods are created. In our example, the EKS cluster ends up with three worker nodes, consisting of one m5d.2xlarge instance and two m5d.xlarge instances. The last ContainerWorker Pod stays in the state of Pending because the number of worker nodes has reached its maximum (3 in our example) and no more worker nodes can be attached.

$ kubectl get pods -n hivemr3
NAME                               READY   STATUS    RESTARTS   AGE
efs-provisioner-649ff654ff-msjzb   1/1     Running   0          42m
hivemr3-hiveserver2-wcr5g          1/1     Running   0          24m
hivemr3-metastore-0                1/1     Running   0          24m
mr3master-8561-0-v9fmp             1/1     Running   0          24m
mr3worker-f2c9-1                   1/1     Running   0          21m
mr3worker-f2c9-2                   1/1     Running   0          18m
mr3worker-f2c9-3                   1/1     Running   0          6m13s
mr3worker-f2c9-4                   1/1     Running   0          4m3s
mr3worker-f2c9-5                   0/1     Pending   0          2m13s

Here is the progress of scale-out operations:

Here is the progress of scale-in operations:

Note that the EKS cluster does not remove all worker nodes because the configuration key mr3.auto.scale.in.min.hosts in mr3-site.xml is set to 1, which means that no scale-in operation is performed if the number of worker nodes is 1.

The user can check the progress of autoscaling from the log of the DAGAppMaster Pod.

$ kubectl log -n hivemr3 mr3master-8561-0-v9fmp -f | grep -e Scale -e Scaling -e average

Deleting the EKS cluster

Because of the additional components configured manually, it take a few extra steps to delete the EKS cluster. In order to delete the EKS cluster (created with eksctl), proceed in the following order.

1. Remove the new inbound rule (using NFS) in the security group for EFS.
2. Delete EFS on the AWS console.
3. Remove inline policies from the IAM roles for the mr3-master and mr3-worker node groups.
4. Execute the command eksctl delete cluster -f eks/cluster.yaml.

$ eksctl delete cluster -f eks/cluster.yaml 

If the last command fails, the user should delete the EKS cluster manually. Proceed in the following order on the AWS console.

1. Delete security groups manually.
2. Delete the NAT gateway created for the EKS cluster, delete the VPC, and then delete the Elastic IP address.
3. Delete the LoadBalancer.
4. Delete IAM roles.
5. Delete CloudFormations.