Running Timeline server

The following file and directories are relevant to running Timeline Server using Helm.

# resources for executing Helm
└── kubernetes
    └── helm
        └── ats
            └── values.yaml

# resources for running Timeline Server
└── kubernetes
    ├── ats-conf
    └── ats-key

The file kubernetes/helm/ats/values.yaml defines the default values for the Helm chart. Typically the user creates another file (e.g., values-ats.yaml) in order to override some of these default values. The user can update configuration files in the directory kubernetes/ats-conf to control the behavior of Timeline Server. The user should copy keytab files to the directory kubernetes/ats-key.

Assuming that a new YAML file values-ats.yaml overrides the default values in kubernetes/helm/ats/values.yaml, the user can run Timeline Server with namespace hivemr3 as follows:

$ ln -s $(pwd)/kubernetes/ats-conf/ kubernetes/helm/ats/conf
$ ln -s $(pwd)/kubernetes/ats-key/ kubernetes/helm/ats/key
$ helm install --namespace hivemr3 kubernetes/helm/ats -f values-ats.yaml

Here the first two commands create symbolic links so that Helm can access the directories kubernetes/ats-conf and kubernetes/ats-key directly.

Enabling SSL

In order to enable SSL, the user should have a TrustStore file. Follow the instruction in Enabling SSL to perform the following:

  • Create a self-signed certificate for SSL by executing the script kubernetes/generate-hivemr3-ssl.sh, and set the environment variable MR3_SSL_KEYSTORE_PASSWORD in kubernetes/config-run.sh.
  • Configure Timeline Server to use SSL.

Then execute the script kubernetes/run-ats.sh with an argument --generate-truststore (which is mandatory) in order to create a KeyStore file kubernetes/ats-key/hivemr3-ssl-certificate.jks.

$ pwd
/home/gla/mr3-run/kubernetes
$ ./run-ats.sh --generate-truststore

Updating kubernetes/helm/ats/values.yaml

After updating the YAML file values-ats.yaml and the configuration files, the user can use Helm in the same way to run Timeline Server. Below we describe each part of kubernetes/helm/ats/values.yaml.

name

This part specifies the names of various Kubernetes objects. It is unnecessary to override the default values.

name:
  ats:
    configMap: hivemr3-ats-configmap
    secret: hivemr3-ats-secret
  envConfigMap: ats-env-configmap
  persistentVolume: workdir-ats-pv
  persistentVolumeClaim: workdir-ats-pvc

docker for the Docker image

docker:
  image: 10.1.91.17:5000/ats-2.7.7:latest
  user: ats
  imagePullPolicy: Always
  imagePullSecrets: 
  • docker/image specifies the full name of the Docker image including a tag.
  • docker/user should match the user specified in kubernetes/ats/Dockerfile when creating the Docker image.
  • docker/imagePullPolicy specifies the pull policy for the Docker image.
  • docker/imagePullSecrets specifies the pull secret for the Docker image. It is set to empty if no pull secret is necessary.

ats for Timeline Server

  secretKey: 
    
  httpPort: 8188
  httpsPort: 8190
  externalIp: 10.1.91.41

  createSecret: true

  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 1
      memory: 4Gi
  heapSize: 2048
  • ats/secretKey specifies the secret key which is used to set the environment variable ATS_SECRET_KEY. If it is set to empty, a random secret key is generated. In order for DAGAppMaster to user Timeline Server, the user should set the field amConfig/atsSecretKey in kubernetes/helm/hive/values.yaml to the same secret key.
  • ats/httpPort and ats/httpsPort specify the port numbers for exposing Timeline Server to the outside of the Kubernetes cluster.
  • ats/externalIp specifies the host for the Service for exposing Timeline Server to the outside of the Kubernetes cluster.
  • ats/createSecret specifies whether or not to crate a Secret from those files in the directory kubernetes/ats-key.
  • ats/atsResources specifies the resources to be allocated to the Timeline Server Pod.
  • ats/heapSize specifies the Java heap size (in MB) for Timeline Server.

workDir for PersistentVolume

workDir:
  isNfs: true
  nfs:
    server: "10.1.91.17"
    path: "/work/nfs/mr3-run-work-dir"
  volumeSize: 10Gi
  volumeClaimSize: 10Gi
  storageClassName: ""
  • workDir/isNfs specifies whether the PersistentVolume uses NFS or not.
  • workDir/nfs/server and workDir/nfs/path specify the address of the NFS server and the path exported by the NFS server (when workDir/isNfs is set to true).
  • workDir/volumeSize and workDir/volumeClaimSize specify the size of the PersistentVolume and the PersistentVolumeClaim.
  • workDir/storageClassName specifies the StorageClass of the PersistentVolume.
  • workDir/volumeStr specifies the PersistentVolume to use when workDir/isNfs is set to false. For example, volumeStr: "hostPath:\n path: /work/nfs/mr3-run-work-dir" creates a hostPath PersistentVolume.

logLevel

logLevel: INFO
  • logLevel specifies the logging level.