🚧 This Article Is Under Construction​

This post currently contains raw experiment results only. A full analysis and discussion will be added soon — please check back later.

Introduction​

In our previous article, we evaluated the performance of Hive 4 on MR3 1.11 and Trino 453 on the 10TB TPC-DS benchmark. The results can be summarized as follows:

• In terms of the total running time, Hive 4 on MR3 runs slightly faster than Trino -- Hive 4 on MR3 5744 seconds vs Trino 5798 seconds.
• In terms of the geometric mean of running times, Trino responds about 15 percent faster than Hive 4 on MR3 -- Trino 17.99 seconds vs Hive 4 on MR3 21.02 seconds.

Introduction​

Recently Apache Hive 4 was released after a hiatus of several years. We have released Hive 4 on MR3 which replaces Tez with MR3 as the default execution engine in Hive 4.

Introduction​

In our previous article, we evaluate the performance of Trino 418 and Hive on MR3 1.7 using the TPC-DS Benchmark with a scale factor of 10TB.

• In terms of the total running time, the two systems are comparable: Trino 7424 seconds vs Hive on MR3 7415 seconds.
• In terms of the geometric mean of running times, Trino is faster than Hive on MR3: Trino 21.75 seconds vs Hive on MR3 27.68 seconds.
• Trino returns wrong answers on query 23 after running for 1756 seconds.
• Trino fails to complete query 72 after running for 156 seconds.

Introduction​

In our previous article, we have shown that Hive on MR3 1.7 runs much faster than Spark 3.4.0 on the TPC-DS benchmark with a scale factor of 10TB (7415 seconds vs 19669 seconds). The performance gap is expected to widen further due to improvements in Hive on MR3 1.8 (6867 seconds vs 7415 seconds). Still, however, there is a category of queries on which Hive on MR3 seems noticeably slower than Spark: single-table queries with no joins.

Introduction​

Before MR3 1.8, Hive on MR3 was built with Java 8. From MR3 1.8, we release Hive on MR3 built with Java 17 as well. An immediate benefit of upgrading to Java 17 is a significant improvement in speed and stability.

Introduction​

In this article, we evaluate the performance of the following systems.

1. Trino 418 (released on May 17, 2023)
2. Spark 3.4.0 (released on Apr 13, 2023)
3. Hive 3.1.3 on MR3 1.7 (released on May 15, 2023)

Introduction​

In this article, we evaluate the performance of the following systems.

1. Spark 2.3.8
2. Spark 3.2.1
3. Hive 3.1.2 on MR3 1.4
4. Hive-LLAP in HDP 3.1.4 (3.1.0.3.1.4.0-315)

Introduction​

MR3 is a general purpose execution engine that provides native support for Hadoop and Kubernetes. While Hive on MR3 is its main application, MR3 can easily execute MapReduce/Tez jobs as well. For example, Hive on MR3 supports compaction without relying on MapReduce because MapReduce jobs performing compaction are directly sent to MR3.

Hive and Presto​

Hive and Presto have developed a tortoise-and-hare story over the past 8 years. Initially conceived at Facebook and open sourced in August 2008, Hive was hailed as a breakthrough in the SQL-on-Hadoop technology and generally regarded as the de facto standard. Then in 2012, Facebook started to develop Presto as a replacement of Hive, which was considered too slow for their daily workload. As Facebook was specific about its goal in developing Presto, the future of Hive did not look so bright.