This page explains additional steps for running MR3-UI and Grafana along with Hive on MR3.

Setting environment variables

To run MR3-UI and Grafana, we should set two environment variables MR3_APPLICATION_ID_TIMESTAMP and ATS_SECRET_KEY before executing HiveServer2.

  • Set MR3_APPLICATION_ID_TIMESTAMP to a positive integer whose last four digits determine the name of the Service for exposing MR3 DAGAppMaster. It should be fixed in advance so that we can configure Prometheus which contacts MR3 DAGAppMaster to collect data (in timeline-conf/prometheus.yml).
  • Set ATS_SECRET_KEY to a random string which is used as a secret key for accessing the Timeline Server. By setting ATS_SECRET_KEY, we ensure that both the Timeline Server and MR3 DAGAppMaster share the same secret key. Then MR3 DAGAppMaster can deliver history logs to the Timeline Server.

Here are examples of setting MR3_APPLICATION_ID_TIMESTAMP and ATS_SECRET_KEY.

# set manually
$ export MR3_APPLICATION_ID_TIMESTAMP=9999
$ export ATS_SECRET_KEY=22f767f8-7c56-421d-ac36-f2cf2392c1ba

# generate random values
$ export MR3_APPLICATION_ID_TIMESTAMP=$RANDOM
$ export ATS_SECRET_KEY=$(cat /proc/sys/kernel/random/uuid)

# get the last four digits of MR3_APPLICATION_ID_TIMESTAMP
$ printf "%04d\n" ${MR3_APPLICATION_ID_TIMESTAMP: -4}
4679

To run MR3-UI and Grafana, we need to check or update the following files:

├── conf
│   ├── mr3-site.xml
│   └── yarn-site.xml
├── timeline-conf
│   ├── configs.env
│   └── prometheus.yml
└── yaml
    ├── timeline-service.yaml
    ├── timeline.yaml
    └── workdir-pv-timeline.yaml

conf/mr3-site.xml

The following configuration keys should be set to true in order for MR3 DAGAppMaster to send data to the Timeline Server and Prometheus.

$ vi mr3-site.xml

<property>
  <name>mr3.app.history.logging.enabled</name>
  <value>true</value>
</property>

<property>
  <name>mr3.dag.history.logging.enabled</name>
  <value>true</value>
</property>

<property>
  <name>mr3.prometheus.enable.metrics</name>
  <value>true</value>
</property>

conf/yarn-site.xml

To use a Timeline Server, the configuration key yarn.timeline-service.enabled should be set to true.

$ vi conf/yarn-site.xml

<property>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>

yaml/timeline-service.yaml

This file creates a Service for exposing MR3-UI (a Jetty server) and Grafana, as well as the Timeline Server, to the outside of the Kubernetes cluster. The user should specify an IP address with a valid host name.

$ vi yaml/timeline-service.yaml
spec:
  externalIPs:
  - 192.168.10.1

yaml/timeline.yaml

$ vi yaml/timeline.yaml

spec:
  template:
    spec:
      hostAliases:
      - ip: "192.168.10.1"
        hostnames:
        - "orange1"
      containers:
      - image: mr3project/mr3ui:1.5   # Timeline Server
      - image: mr3project/mr3ui:1.5   # MR3-UI
      - image: mr3project/mr3ui:1.5   # Prometheus
      - image: mr3project/mr3ui:1.5   # Grafana
  • The spec/hostAliases field lists aliases for hosts that may not be found in the default DNS. It should include the host assigned to the Service created in the previous step.
  • The image field in the spec/containers section specifies the Docker image for MR3-UI and Grafana.

yaml/workdir-pv-timeline.yaml

This file creates a PersistentVolume for storing data for MR3-UI and Prometheus. The user should update it in order to use a desired type of PersistentVolume. The PersistentVolume should be writable to the user with UID 1000 (which is specified in kubernetes/ats/Dockerfile).

$ vi yaml/workdir-pv-timeline.yaml
spec:
  nfs:
    server: "192.168.10.1"
    path: "/home/nfs/hivemr3"

To reuse the PersistentVolume created by yaml/workdir-pv.yaml, use the PersistentVolumeClaim workdir-pvc and update run-timeline.sh to skip the creation of a new PersistentVolume for MR3-UI/Grafana.

$ vi yaml/timeline.yaml

spec:
  template:
    spec:
      volumes:
      - name: work-dir-volume
        persistentVolumeClaim:
          claimName: workdir-pvc

timeline-conf/configs.env

This file configures MR3-UI to specify the address of the Timeline Server. Since MR3-UI pages are the output of a JavaScript application running on the client side, the timeline field should be a valid address outside the Kubernetes cluster. Set it to use the host assigned to the Service created with yaml/timeline-service.yaml.

$ vi timeline-conf/configs.env

ENV = {
  hosts: {
    timeline: "http://orange1:9188/"
  },
};

timeline-conf/prometheus.yml

This file configures the Prometheus server. The field static_configs/targets should specify the address of MR3 DAGAppMaster, so use the last four digits of MR3_APPLICATION_ID_TIMESTAMP (e.g., 9999 in service-master-9999-0).

$ vi timeline-conf/prometheus.yml

    static_configs:
      - targets: ["service-master-9999-0.hivemr3.svc.cluster.local:9890", "service-master-9999-0.hivemr3.svc.cluster.local:9890"]

Running MR3-UI and Grafana

In order to run MR3-UI and Grafana, the user can execute the script run-timeline.sh.

$ ./run-metastore.sh 
$ ./run-hive.sh 

$ ./run-timeline.sh 
Error from server (AlreadyExists): namespaces "hivemr3" already exists
persistentvolume/workdir-pv-timeline created
persistentvolumeclaim/workdir-pvc-timeline created
Error from server (AlreadyExists): secrets "env-secret" already exists
configmap/hivemr3-timeline-conf-configmap created
secret/hivemr3-timeline-secret created
ATS_SECRET_KEY=7ea5c6fc-62fb-4456-9855-aec5006ecaeb
configmap/client-timeline-config created
statefulset.apps/hivemr3-timeline created
service/timeline created

The DAGAppMaster Pod may restart a few times if the MR3-UI/Grafana Pod does not start quickly (because no Timeline Server can be contacted). The MR3-UI/Grafana Pod runs four containers.

$ kubectl get pods -n hivemr3
NAME                                 READY   STATUS    RESTARTS   AGE
hivemr3-hiveserver2-b6889c9d-98dsp   1/1     Running   0          2m28s
hivemr3-metastore-0                  1/1     Running   0          2m31s
hivemr3-timeline-0                   4/4     Running   0          2m17s
mr3master-4633-0-8495b98f7f-c9792    1/1     Running   0          2m15s

Accessing MR3-UI and Grafana

The ports assigned to MR3-UI (a Jetty server) and Grafana can be found in yaml/timeline-service.yaml.

$ vi yaml/timeline-service.yaml

spec:
  ports:
  - name: mr3-ui-jetty
    protocol: TCP
    port: 8080
    targetPort: 8080
  - name: grafana
    port: 3000
    targetPort: 3000

In our example, the host assigned to the Service for MR3-UI and Grafana is orange1, so we access MR3-UI and Grafana at the following URLs.

  • MR3-UI: http://orange1:8080
  • Grafana: http://orange1:3000

MR3-UI shows details of DAGs executed by MR3 for Hive.

typescript-mr3ui-fs8

For Grafana, the password for the user admin is initialized to admin, and can be changed after the first login. The user can watch MR3 for Hive on the dashboard MR3 for Hive.

typescript-grafana-dashboards-fs8

Calendar Feb 22, 2023