With MR3-UI and Grafana
This page explains additional steps for running MR3-UI and Grafana along with Hive on MR3.
If install.sh
was not executed while installing Hive on MR3,
manually create a symbolic link to the directory kubernetes/timeline-conf
.
ln -s ../../kubernetes/timeline-conf/ timeline/conf
The file timeline/values.yaml
defines the default values for the Helm chart.
Typically the user creates another YAML file to override some of these default values.
In our example, we create a new YAML file timeline/values-timeline.yaml
.
To run MR3-UI and Grafana, we need to check or update the following files.
├── hive/conf
│ └── mr3-site.xml
└── timeline/conf
└── prometheus.yml
Setting environment variables
To run Grafana, we should fix the values for the environment variable
MR3_APPLICATION_ID_TIMESTAMP
.
- Set
MR3_APPLICATION_ID_TIMESTAMP
to a positive integer whose last four digits determine the name of the Service for exposing MR3 DAGAppMaster. It should be fixed in advance so that we can configure Prometheus which contacts MR3 DAGAppMaster to collect data (intimeline/conf/prometheus.yml
).
When using Helm,
we specify a value for the environment variable in values-hive.yaml
,
as shown in the example below.
- The field
amConfig.timestamp
inhive/values-hive.yaml
setsMR3_APPLICATION_ID_TIMESTAMP
.
vi hive/values-hive.yaml
amConfig:
timestamp: 9999
Basic settings
Open timeline/values-timeline.yaml
and set the following fields.
vi values-timeline.yaml
docker:
image: mr3project/mr3ui:1.5
timeline:
externalIp: 192.168.10.1
hostAliases:
- ip: "192.168.10.1"
hostnames:
- "orange1"
docker.image
specifies the full name of the Docker image including a tag. We use the pre-built Docker imagemr3project/mr3ui:1.5
.timeline.externalIp
specifies the host for the Service for exposing MR3-UI and Grafana to the outside of the Kubernetes cluster. The user should specify an IP address with a valid host name.hostAliases
lists aliases for hosts that may not be found in the default DNS. It should include the host assigned to the Service for exposing MR3-UI and Grafana.
PersistentVolumeClaim
Grafana uses the PersistentVolumeClaim workdir-pvc
created for Hive on MR3.
The PersistentVolume should be writable to the user with UID 1000.
To use a local directory inside the Docker container instead
(e.g., if PersistentVolumeClaim is not created for Hive on MR3),
set the field workDir.use
to false.
vi timeline/values-timeline.yaml
workDir:
use: false
hive/conf/mr3-site.xml
The following configuration keys should be set to true in order for MR3 DAGAppMaster to send data to 1) the Timeline Server for MR3-UI and 2) Prometheus.
vi hive/conf/mr3-site.xml
<property>
<name>mr3.app.history.logging.enabled</name>
<value>true</value>
</property>
<property>
<name>mr3.dag.history.logging.enabled</name>
<value>true</value>
</property>
<property>
<name>mr3.ui.create.server</name>
<value>true</value>
</property>
<property>
<name>mr3.prometheus.enable.metrics</name>
<value>true</value>
</property>
By default,
MR3 DAGAppMaster uses the PersistentVolumeClaim workdir-pvc
to store the database for its internal Timeline Server.
Update the following configuration key to change the directory for storing the database
(e.g., /tmp/leveldb
inside the Docker image).
vi hive/conf/mr3-site.xml
<property>
<name>mr3.ui.timeline-service.leveldb-timeline-store.path</name>
<value>/opt/mr3-run/work-dir/leveldb</value>
</property>
timeline/conf/prometheus.yml
This file configures the Prometheus server.
The field static_configs/targets
should specify the address of MR3 DAGAppMaster,
so use the last four digits of MR3_APPLICATION_ID_TIMESTAMP
(e.g., 9999
in service-master-9999-0
).
vi timeline/conf/prometheus.yml
static_configs:
- targets: ["service-master-9999-0.hivemr3.svc.cluster.local:9890", "service-master-9999-0.hivemr3.svc.cluster.local:9890"]
Running MR3-UI and Grafana
Assuming that
a new YAML file timeline/values-timeline.yaml
overrides the default values in timeline/values.yaml
,
the user can run MR3-UI and Grafana with namespace hivemr3
as follows.
helm install --namespace hivemr3 timeline -f timeline/values-timeline.yaml
Then the user can execute Metastore and HiveServer2.
Accessing MR3-UI and Grafana
The ports assigned to MR3-UI and Grafana can be found in
timeline/values.yaml
.
vi yaml/mr3ui-service.yaml
ui:
httpPort: 19003
grafana:
httpPort: 3000
In our example,
the host assigned to the Service for MR3-UI and Grafana is orange1
,
so we access MR3-UI and Grafana at the following URLs.
- MR3-UI:
http://orange1:19003
- Grafana:
http://orange1:3000
MR3-UI shows details of DAGs executed by MR3 for Hive.
For Grafana,
the password for the user admin
is initialized to admin
,
and can be changed after the first login.
The user can watch MR3 for Hive on the dashboard MR3 for Hive
.