Accessing S3 with SSL

In order to access S3 with SSL enabled, the user should set the configuration key fs.s3a.connection.ssl.enabled in conf/core-site.xml.

vi conf/core-site.xml

<property>
  <name>fs.s3a.connection.ssl.enabled</name>
  <value>true</value>
</property>

For accessing custom S3-compatible storage, the address for the storage server should be revised to use HTTPS.

vi conf/core-site.xml

<property>
  <name>fs.s3a.endpoint</name>
  <value>https://my.s3.server.address:9000</value>
</property>

Next the user should make a copy of the certificate file for connecting to the storage server and set MR3_S3_CERTIFICATE in config-run.sh to point to the copy.

vi config-run.sh

ENABLE_SSL=true
...

MR3_S3_CERTIFICATE=/home/hive/mr3/kubernetes/s3-public.cert

Now executing run-hive.sh adds the certificate to the KeyStore file key/hivemr3-ssl-certificate.jks, and every component that uses the KeyStore file can access S3. For example, HiveServer2 can access S3 because the configuration key hive.server2.keystore.path points to the KeyStore file.

vi conf/hive-site.xml

<property>
  <name>hive.server2.keystore.path</name>
  <value>/opt/mr3-run/key/hivemr3-ssl-certificate.jks</value>
</property>

For DAGAppMaster and ContainerWorkers, the user should check if Java properties javax.net.ssl.trustStore and javax.net.ssl.trustStoreType are properly set in their command-line options:

vi conf/mr3-site.xml

<property>
  <name>mr3.am.launch.cmd-opts</name>
  <value>... -Djavax.net.ssl.trustStore=/opt/mr3-run/key/hivemr3-ssl-certificate.jks -Djavax.net.ssl.trustStoreType=jks</value>
</property>

<property>
  <name>mr3.container.launch.cmd-opts</name>
  <value>... -Djavax.net.ssl.trustStore=/opt/mr3-run/key/hivemr3-ssl-certificate.jks -Djavax.net.ssl.trustStoreType=jks</value>
</property>