For running HiveCLI, the user should use HiveCLI included in the MR3 release.

Running HiveCLI

In order to run HiveCLI, set the following environment variables in env.sh as necessary:

HIVE_CLIENT_HEAPSIZE=1024

USER_PRINCIPAL=hive@HADOOP
USER_KEYTAB=/home/hive/hive.keytab
  • HIVE_CLIENT_HEAPSIZE specifies the heap size (in MB) for HiveCLI.
  • USER_PRINCIPAL and USER_KEYTAB specify the principal and keytab file for the user executing HiveCLI in a secure cluster with Kerberos.

In order to start a HiveCLI session, execute hive/run-hive-cli.sh with the following options:

--local                   # Run jobs with configurations in conf/local/ (default).
--cluster                 # Run jobs with configurations in conf/cluster/.
--tpcds                   # Run jobs with configurations in conf/tpcds/.
--hivesrc3                # Choose hive3-mr3 (based on Hive 3.1.3)
--hivesrc4                # Choose hive4-mr3 (based on Hive 4.0.0, default)
--amprocess               # Run the MR3 DAGAppMaster in LocalProcess mode.
--hiveconf <key>=<value>  # Add a configuration key/value; may be repeated at the end.
<HiveCLI option>          # Add a HiveCLI option; may be repeated at the end.

With --amprocess, HiveCLI runs a new MR3 DAGAppMaster in LocalProcess mode. Hence a new process starts on the same machine where the script is run. The user can append as many HiveCLI options (for the command hive from Hive) as necessary to the command. (Currently --amprocess cannot be used in a secure cluster with Kerberos; see the documentation on LocalProcess mode.)

In a secure cluster with Kerberos, HiveCLI uses the Kerberos ticket provided by the user to authenticate itself to Yarn. Hence the Kerberos ticket should be valid at the time of executing the script. In a non-secure cluster without Kerberos, HiveCLI communicates with Yarn as the user executing the script.

Output directory of HiveCLI

Executing hive/run-hive-cli.sh creates a new directory under hive/run-hive-cli-result:

hive-mr3--2018-03-14--00-13-40-8f4aaf03/
├── command
├── conf
│   ├── beeline-log4j.properties
...
│   └── yarn-site.xml
├── env
├── hive-logs
│   └── hive.log
└── out.txt

The name of the HiveCLI output directory ends with a random sequence such as 8f4aaf03.

  • command contains the command executed to start the HiveCLI session.
  • conf is a directory containing all configuration files that are effective at the time of starting the HiveCLI session.
  • env lists all environment variables that are effective at the time of starting the HiveCLI session.
  • hive-logs/hive.log is the log file for HiveCLI.
  • out.txt is the output of hive/run-hive-cli.sh.
Calendar Jul 22, 2024