For DAGAppMasters and ContainerWorkers of MR3, Hive on MR3 uses by default a logging configuration file included in the MR3 release. For Hive 2 and later, it is mr3-container-log4j2.properties. Below we describe mr3-container-log4j2.properties.

Using mr3-container-log4j2.properties

status = INFO
name = MR3ContainerLog4j2

property.mr3.root.logger = INFO
property.mr3.log.dir = ${sys:yarn.app.container.log.dir}
property.mr3.log.file = run.log

appenders = console, DRFA

appender.console.type = Console
appender.console.name = consoleLogger
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d [%t] %-5p %c %x - %m%n

appender.DRFA.type = RollingRandomAccessFile
appender.DRFA.name = DRFALogger
appender.DRFA.immediateFlush = true
appender.DRFA.fileName = ${sys:mr3.log.dir}/${sys:mr3.log.file}
appender.DRFA.filePattern = ${sys:mr3.log.dir}/${sys:mr3.log.file}.%d{yyyy-MM-dd}.%i
appender.DRFA.layout.type = PatternLayout
appender.DRFA.layout.pattern = %d [%t] %-5p %c %x - %m%n
appender.DRFA.policies.type = Policies
appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
appender.DRFA.policies.time.interval = 1
appender.DRFA.policies.time.modulate = true
appender.DRFA.policies.size.type = SizeBasedTriggeringPolicy
appender.DRFA.policies.size.size = 250MB
appender.DRFA.strategy.type = DefaultRolloverStrategy
appender.DRFA.strategy.max = 30

rootLogger.level = ${sys:mr3.root.logger}
rootLogger.appenderRef.file.ref = DRFALogger
rootLogger.appenderRef.console.ref = consoleLogger
rootLogger.appenderRef.console.level = FATAL
rootLogger.includeLocation = false

By default, ${sys:mr3.root.logger} expands to INFO, and ${sys:yarn.app.container.log.dir} expands to a logging directory determined by Yarn at runtime.

The user can easily change the logging level by resetting the environment variable LOG_LEVEL in env.sh:

LOG_LEVEL=INFO

Using a differnt logging configuration file

The user can also use a different logging configuration file in two steps. First the user should update command-line options for DAGAppMasters and ContainerWorkers. In mr3-site.xml, mr3.am.launch.cmd-opts is the key for specifying command-line options for DAGAppMasters, and mr3.container.launch.cmd-opts is the key for specifying command-line options for ContainerWorkers. Thus the user should append a new command-line option -Dlog4j.configurationFile=<name of the new logging configuration file> to the values for these keys, as shown in the following example:

<property>
  <name>mr3.am.launch.cmd-opts</name>
  <value>-server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configurationFile=custom-log4j.properties</value>
  <description>
    Command-line options for launching DAGAppMaster
  </description>
</property>
<property>
  <name>mr3.container.launch.cmd-opts</name>
  <value>-server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configurationFile=custom-log4j.properties</value>
  <description>
    Command-line options for launching ContainerWorkers
  </description>
</property>

Second the user should register the new logging configuration file as a local resource so that it appears in the classpath. Here is an example of registering a logging configuration file custom-log4j.properties as a local resource:

  • copy the file to /tmp/custom-log4j.properties on HDFS.
  • append the path /tmp/custom-log4j.properties to the value of mr3.aux.uris in mr3-site.xml: 
    <property>
  <name>mr3.aux.uris</name>
  <value>${auxuris},/tmp/custom-log4j.properties</value>
  <description>
    URIs for the MR3 auxiliary jar files
  </description>
</property>

Similarly to the one included in the MR3 release, the new logging configuration file may use ${sys:mr3.root.logger} and ${sys:yarn.app.container.log.dir} which automatically expand at runtime.

Calendar May 24, 2023