With MR3-UI and Grafana

This page explains additional steps for running MR3-UI and Grafana along with Hive on MR3.

caution

If install.sh was not executed while installing Hive on MR3, manually create a symbolic link to the directory kubernetes/timeline-conf.

ln -s ../../kubernetes/timeline-conf/ timeline/conf

The file timeline/values.yaml defines the default values for the Helm chart. Typically the user creates another YAML file to override some of these default values. In our example, we create a new YAML file timeline/values-timeline.yaml.

To run MR3-UI and Grafana, we need to check or update the following files.

├── hive/conf
│   └── mr3-site.xml
└── timeline/conf
    └── prometheus.yml

Setting environment variables

To run Grafana, we should fix the values for the environment variable MR3_APPLICATION_ID_TIMESTAMP.

• Set MR3_APPLICATION_ID_TIMESTAMP to a positive integer whose last four digits determine the name of the Service for exposing MR3 DAGAppMaster. It should be fixed in advance so that we can configure Prometheus which contacts MR3 DAGAppMaster to collect data (in timeline/conf/prometheus.yml).

When using Helm, we specify a value for the environment variable in values-hive.yaml, as shown in the example below.

• The field amConfig.timestamp in hive/values-hive.yaml sets MR3_APPLICATION_ID_TIMESTAMP.

vi hive/values-hive.yaml

amConfig:
  timestamp: 9999

Basic settings

Open timeline/values-timeline.yaml and set the following fields.

vi values-timeline.yaml

docker:
  image: mr3project/mr3ui:1.5

timeline:
  externalIp: 192.168.10.1

hostAliases:
- ip: "192.168.10.1"
  hostnames:
  - "orange1"

• docker.image specifies the full name of the Docker image including a tag. We use the pre-built Docker image mr3project/mr3ui:1.5.
• timeline.externalIp specifies the host for the Service for exposing MR3-UI and Grafana to the outside of the Kubernetes cluster. The user should specify an IP address with a valid host name.
• hostAliases lists aliases for hosts that may not be found in the default DNS. It should include the host assigned to the Service for exposing MR3-UI and Grafana.

PersistentVolumeClaim

Grafana uses the PersistentVolumeClaim workdir-pvc created for Hive on MR3. The PersistentVolume should be writable to the user with UID 1000.

To use a local directory inside the Docker container instead (e.g., if PersistentVolumeClaim is not created for Hive on MR3), set the field workDir.use to false.

vi timeline/values-timeline.yaml

workDir:
  use: false

hive/conf/mr3-site.xml

The following configuration keys should be set to true in order for MR3 DAGAppMaster to send data to 1) the Timeline Server for MR3-UI and 2) Prometheus.

vi hive/conf/mr3-site.xml

<property>
  <name>mr3.app.history.logging.enabled</name>
  <value>true</value>
</property>

<property>
  <name>mr3.dag.history.logging.enabled</name>
  <value>true</value>
</property>

<property>
  <name>mr3.ui.create.server</name>
  <value>true</value>
</property>

<property>
  <name>mr3.prometheus.enable.metrics</name>
  <value>true</value>
</property>

By default, MR3 DAGAppMaster uses the PersistentVolumeClaim workdir-pvc to store the database for its internal Timeline Server. Update the following configuration key to change the directory for storing the database (e.g., /tmp/leveldb inside the Docker image).

vi hive/conf/mr3-site.xml

<property>
  <name>mr3.ui.timeline-service.leveldb-timeline-store.path</name>
  <value>/opt/mr3-run/work-dir/leveldb</value>
</property>

timeline/conf/prometheus.yml

This file configures the Prometheus server. The field static_configs/targets should specify the address of MR3 DAGAppMaster, so use the last four digits of MR3_APPLICATION_ID_TIMESTAMP (e.g., 9999 in service-master-9999-0).

vi timeline/conf/prometheus.yml

    static_configs:
      - targets: ["service-master-9999-0.hivemr3.svc.cluster.local:9890", "service-master-9999-0.hivemr3.svc.cluster.local:9890"]

Running MR3-UI and Grafana

Assuming that a new YAML file timeline/values-timeline.yaml overrides the default values in timeline/values.yaml, the user can run MR3-UI and Grafana with namespace hivemr3 as follows.

helm install --namespace hivemr3 timeline -f timeline/values-timeline.yaml

Then the user can execute Metastore and HiveServer2.

Accessing MR3-UI and Grafana

The ports assigned to MR3-UI and Grafana can be found in timeline/values.yaml.

vi yaml/mr3ui-service.yaml

ui:
  httpPort: 19003

grafana:
  httpPort: 3000

In our example, the host assigned to the Service for MR3-UI and Grafana is orange1, so we access MR3-UI and Grafana at the following URLs.

• MR3-UI: http://orange1:19003
• Grafana: http://orange1:3000

MR3-UI shows details of DAGs executed by MR3 for Hive.

For Grafana, the password for the user admin is initialized to admin, and can be changed after the first login. The user can watch MR3 for Hive on the dashboard MR3 for Hive.