Prometheus for DAGAppMster

DAGAppMaster can run a Prometheus client to report the statistics on DAGs as well as the states of Java VM.

• If the configuration key mr3.prometheus.enable.metrics is set to true in mr3-site.xml, DAGAppMaster runs a Prometheus client which creates an HTTP server on port specified by the configuration key mr3.prometheus.httpserver.port.
• By default, the Prometheus client exports metrics on DAGs only.
• The user can export metrics on Java VM as well (from io.prometheus.client.hotspot.DefaultExports) by setting the configuration key mr3.prometheus.enable.jvm.metrics to true.
• DAGAppMaster updates the metrics at an interval specified by the configuration key mr3.check.memory.usage.event.interval.secs.

The following table shows all the metrics on DAGs. Except for the metric mr3_dags and those starting with mr3_dags_total, every metric is associated with a ContainerGroup whose ID is used as its label. For example, a metric mr3_heap_bytes_max reports the aggregate size of the Java heap of ContainerWorkers belonging to a particular ContainerGroup. Hence, if multiple ContainerGroups coexist, the total size of the Java heap of all ContainerWorkers can be obtained as the sum of all such metrics. In particular, if the configuration key mr3.container.stop.cross.dag.reuse is set to true, a new metric is exported for each DAG which creates its own ContainerGroup.

Creating a Service on Kubernetes

On Kubernetes, the user may want to create a Service for exposing the HTTP server of DAGAppMaster to the outside of the Kubernetes cluster. Without such a Service, the Prometheus server should run inside the Kubernetes cluster and use the IP address of the DAGAppMaster Pod assigned by Kubernetes.

Creating a Service for exposing the HTTP server of DAGAppMaster takes two steps. First the user should set the environment variable CREATE_PROMETHEUS_SERVICE to true in kubernetes/env.sh.

CREATE_PROMETHEUS_SERVICE=true

Now the script kubernetes/run-hive.sh creates a Service from the YAML file kubernetes/yaml/prometheus-service.yaml. By default, it creates a Service of type LoadBalancer, but the user can customize the Service by updating the YAML file. For example, the user can use a different public IP address by changing the field spec/externalIPs. (It is okay to use the same public IP address that HiveServer2 Service uses as long as a different port is specified.) Second the user should add a label hivemr3_aux=prometheus to the DAGAppMaster Pod by setting the configuration key mr3.k8s.master.pod.additional.labels in kubernetes/conf/mr3-site.xml.

<property>
  <name>mr3.k8s.master.pod.additional.labels</name>
  <value>hivemr3_aux=prometheus</value>
</property>

Now every request to the Service is routed to the DAGAppMaster Pod.

With Helm, the first step can be completed by overriding the default values in kubernetes/helm/hive/values.yaml. The field prometheus/create should be set to true in order to create a Service for Prometheus.

prometheus:
  create: true
  port: 9890
  externalIp: 12.34.56.78

Prometheus for ContainerWorkers

A ContainerWorker can run a Prometheus client to report the statistics on TaskAttempts as well as the states of Java VM.

• If the configuration key mr3.prometheus.worker.enable.metrics is set to true in mr3-site.xml, a ContainerWorker runs a Prometheus client which creates an HTTP server on port specified by the configuration key mr3.prometheus.worker.httpserver.port.
• By default, the Prometheus client exports metrics on TaskAttempts only.
• The user can export metrics on Java VM as well (from io.prometheus.client.hotspot.DefaultExports) by setting the configuration key mr3.prometheus.worker.enable.jvm.metrics to true.

The following table shows all the metrics on TaskAttempts. Every metric uses the ID of the ContainerWorker as its label.

On Yarn, if multiple ContainerWorkers can run on the same node, the configuration key mr3.prometheus.worker.httpserver.port should be set to the default value of 0 so that each ContainerWorker can choose a valid port at runtime. On Kubernetes, it can be set to a fixed port number because each ContainerWorker Pod is assigned a unique IP address.

On Kubernetes, the user can create a Service for exposing the HTTP server of a specific ContainerWorker Pod to the outside of the Kubernetes cluster. Every ContainerWorker Pod has a label mr3-worker-pod-id whose value is unique during the lifetime of the DAGAppMaster Pod. For example, a ContainerWorker Pod mr3worker-8512-4 has a value of 4 for the label.

$ kubectl describe pod -n hivemr3 mr3worker-8512-4 | grep mr3-worker-pod-id
              mr3-worker-pod-id=4

We can create a Service of type LoadBalancer dedicated to the ContainerWorker Pod as follows:

$ vi prometheus-worker-service.yaml 

apiVersion: v1
kind: Service
metadata:
  namespace: hivemr3
  name: prometheus
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 9890
    targetPort: 9890
  selector:
    mr3-worker-pod-id: "4"
  externalIPs:
  - 192.168.10.1

$ kubectl create -f prometheus-worker-service.yaml 
service/prometheus created

Now every request to the Service is routed to the ContainerWorker Pod mr3worker-8512-4.