By running Ranger, we complete a Kubernetes cluster which supports data security and is thus ready for production use:
We can also run Ranger as a separate application (independently of Hive on MR3). If a Docker image for Ranger has already been built, download an MR3 release containing the executable scripts and skip the instruction below for building a Docker image.
$ git clone https://github.com/mr3project/mr3-run-k8s.git
$ cd mr3-run-k8s/
Set the environment variable HIVE_MYSQL_DRIVER
in env.sh
to a MySQL connector jar file compatible with the MySQL database for Ranger.
$ vi env.sh
HIVE_MYSQL_DRIVER=/usr/share/java/mysql-connector-java.jar
Building a Docker image for Ranger
To build a Docker image for Ranger,
go to the installation directory of Hive on MR3 (see Installing on Kubernetes)
and collect all necessary files in the directory kubernetes/ranger
by executing the script build-k8s-ranger.sh
.
$ ./build-k8s-ranger.sh
downloading Solr at /tmp/solr-7.7.2.tgz
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36:18 min
[INFO] Finished at: 2020-10-04T20:14:21+09:00
[INFO] ------------------------------------------------------------------------
build-k8s-ranger.sh
downloads a binary distribution of Apache Solr (version 7.7.2 by default)
and clones a git repository of Ranger (version 2.1.0 by default) under the directory /tmp/ranger
.
Then it compiles Ranger and populates two directories:
kubernetes/ranger
for building a Docker image for Ranger
and kubernetes/hive
for installing the Hive plugin for Ranger.
For building Ranger 2.1.0, the user should use Maven 3.6.2 and Python 3 with the requests
library.
If build-k8s-ranger.sh
fails, check the variable SOLR_DOWNLOAD_URL
(which specifies the address of the binary distribution of Apache Solr)
in the function build_k8s_ranger
of common-build-setup.sh
.
The user can also change the version of Ranger and the directory $RANGER_HOME
where Ranger is built.
If build-k8s-ranger.sh
fails at the last step when building a Ranger distribution,
manually execute the Maven command in the directory $RANGER_HOME
.
function build_k8s_ranger {
...
TMP_DIR=${RANGER_TMP_DIR:-/tmp}
...
SOLR_DOWNLOAD_URL=https://archive.apache.org/dist/lucene/solr/7.7.2/solr-7.7.2.tgz
...
RANGER_HOME=$TMP_DIR/ranger
...
RANGER_GIT_REV=5ec9fbd0b78595084dc847f7fdc9da0506f6c482
RANGER_VERSION=2.1.0
Similarly to building a Docker image for running Hive on MR3,
the user should set an environment variable DOCKER_RANGER_IMG
in kubernetes/env.sh
:
DOCKER_RANGER_IMG=10.1.91.17:5000/ranger:latest
DOCKER_RANGER_IMG
specifies the full name of the Docker image (including a tag) for running Ranger which may include the address of a running Docker server.
The last step is to build a Docker image from Dockerfile in the directory kubernetes/ranger
by executing kubernetes/build-ranger.sh
.
The script builds a Docker image (which contains everything for running Ranger)
and registers it to the Docker server specified in kubernetes/env.sh
.
If successful, the user can pull the Docker image on another node:
$ docker pull 10.1.91.17:5000/ranger
Using default tag: latest
Trying to pull repository 10.1.91.17:5000/ranger ...
latest: Pulling from 10.1.91.17:5000/ranger
...
Status: Downloaded newer image for 10.1.91.17:5000/ranger:latest
Note that after building the Docker image for Ranger, the user should rebuild the Docker image for running Hive on MR3 because HiveServer2 needs the Hive plugin for Ranger.
The user can run Ranger with or without Helm. To run Ranger with Helm, see Running Apache Ranger as well. Below we describe how to run Ranger without Helm.
Configuring the Pod for Ranger
The following files specify how to configure Kubernetes objects for Ranger:
└── kubernetes
├── env.sh
├── ranger-key
└── yaml
├── namespace.yaml
├── ranger-service.yaml
├── ranger.yaml
├── workdir-pvc-ranger.yaml
└── workdir-pv-ranger.yaml
We assume that Ranger belongs to the same namespace as HiveServer2, and reuse namespace.yaml
.
Ranger uses workdir-pvc-ranger.yaml
and workdir-pv-ranger.yaml
which can be configured similarly to workdir-pvc.yaml
and workdir-pv.yaml
.
The PersistentVolume should be writable to the user with UID 1000
(which is specified in kubernetes/ranger/Dockerfile
).
The user should set the following environment variable in kubernetes/env.sh
.
CREATE_RANGER_SECRET=true
CREATE_RANGER_SECRET
specifies whether or not to create a Secret from keytab files in the directorykubernetes/ranger-key
. Usually it should be set to true whether Kerberos is used for authentication or not (because ofkubernetes/ranger-key/install.properties
).
ranger-service.yaml
This file creates a Service for exposing Ranger to the outside of the Kubernetes cluster.
The user should specify an IP address with a valid host name and three port numbers for Ranger
so that both the administrator user from the outside and HiveServer2 from the inside can connect to it using the host name.
Usually there is no need to change the three targetPort
fields which specify port numbers internal to the Ranger Pod.
ports:
- name: ranger-admin-http
protocol: TCP
port: 6080
targetPort: 6080
- name: ranger-admin-https
protocol: TCP
port: 6182
targetPort: 6182
- name: solr
protocol: TCP
port: 6083
targetPort: 6083
externalIPs:
- 10.1.91.41
The sample file in the MR3 release uses 10.1.91.41:6080 as the HTTP address and 10.1.91.41:6182 as the HTTPS address of Ranger. Another address 10.1.91.41:6083 is reserved for the internal communication between Ranger and Solr. The user should make sure that the IP address exists and is not already taken.
ranger.yaml
This file creates a Pod for running Ranger.
Internally the Pod runs two containers (for Ranger itself and for Solr) in parallel.
The user should update the spec/hostAliases
field and the spec/containers
section.
- The
spec/hostAliases
field lists aliases for hosts that may not be found in the default DNS. Usually it suffices to include three hosts: 1) the host running MySQL for Ranger outside the Kubernetes cluster; 2) the host running HiveServer2 inside the Kubernetes cluster; 3) the host running Ranger inside the Kubernetes cluster. In the sample file in the MR3 release,indigo0
is the host running MySQL for Ranger andindigo20
is the host name assigned to HiveServer2 and Ranger.hostAliases: - ip: "10.1.91.17" hostnames: - "indigo0" - ip: "10.1.91.41" hostnames: - "indigo20"
- The
image
field in thespec/containers
section should match the Docker image specified byDOCKER_RANGER_IMG
inkubernetes/env.sh
. - The
resources/requests
andresources/limits
specify the resources to to be allocated to the Ranger container and the Solr container. - The
ports/containerPort
fields should match the port numbers specified in thetargetPort
fields inranger-service.yaml
.
spec:
containers:
- image: 10.1.91.17:5000/ranger
name: solr
resources:
requests:
cpu: 1
memory: 4Gi
limits:
cpu: 1
memory: 4Gi
ports:
- containerPort: 6083
protocol: TCP
- image: 10.1.91.17:5000/ranger
name: ranger
resources:
requests:
cpu: 1
memory: 4Gi
limits:
cpu: 1
memory: 4Gi
ports:
- containerPort: 6080
protocol: TCP
- containerPort: 6182
protocol: TCP
Configuring Ranger
The following files specify how to configure Ranger:
└── kubernetes
├── env.sh
├── ranger-conf
│ ├── core-site.xml
│ ├── krb5.conf
│ ├── ranger-log4j.properties
│ ├── solr-core.properties
│ ├── solr-elevate.xml
│ ├── solr-log4j2.xml
│ ├── solr-managed-schema
│ ├── solr-security.json
│ ├── solr-solrconfig.xml
│ └── solr-solr.xml
└── ranger-key
├── install.properties
└── solr.in.sh
Because of lack of detailed documentation on configuring Ranger, the user is advised to run Ranger initially with minimal changes to the configuration files in the MR3 release (which should work okay in a typical Kubernetes cluster). After getting Ranger up and running, the user can incrementally adjust the configuration to suit particular needs. Otherwise the user might have to fix the configuration by reading the source code of Ranger.
We recommend the user to create three Kerberos keytab files with the following names.
In our example, we assume that indigo20
is the host name assigned to Ranger and that RED
is the Kerberos realm.
The user should copy the keytab files in the directory kubernetes/ranger-key
.
rangeradmin.keytab
with service principalrangeradmin/indigo20@RED
. The instance (e.g.,indigo20
) must match the host name for Ranger.spnego.service.keytab
with service principalHTTP/indigo20@RED
. The primary must beHTTP
and the instance (e.g.,indigo20
) must match the host name for Ranger.rangerlookup.keytab
with principalrangerlookup@RED
. An ordinary principal (without an instance) is okay to use.
Below we describe those sections that are specific to each Kubernetes cluster. To enable SSL, see Enabling SSL. For more details, we refer the reader to the documentation on Ranger.
install.properties
-
SQL_CONNECTOR_JAR
should match the MySQL connector jar file fromHIVE_MYSQL_DRIVER
inenv.sh
.SQL_CONNECTOR_JAR=/opt/mr3-run/lib/mysql-connector-java.jar
If the Docker image for Ranger does not contain a MySQL connector, or a different MySQL connector should be used, the user can copy a MySQL connector jar file to a subdirectory of the PersistentVolume and set
SQL_CONNECTOR_JAR
to point to the file (e.g.,SQL_CONNECTOR_JAR=/opt/mr3-run/ranger/work-dir/lib/mysql-connector-java-8.0.12.jar
). In this way, Ranger can use the custom MySQL connector provided by the user. If the Docker image for Ranger already contains a MySQL connector, the PersistentVolume is not used. -
db_root_user
anddb_root_password
should be set to the ID and password of the root user of MySQL for Ranger.db_root_user=root db_root_password=passwd
-
db_host
should be set to the IP address or the host name of MySQL for Ranger.db_host=indigo0
-
RANGER_ADMIN_LOG_DIR
specifies the directory for logging. By default, Ranger uses a local directory mounted with an emptyDir volume.RANGER_ADMIN_LOG_DIR=/opt/mr3-run/ranger/work-local-dir/log/ranger-admin
solr.in.sh
The environment variable SOLR_AUTHENTICATION_OPTS
should use
the host running Ranger,
the service principal, and service keytab
corresponding to ranger.spnego.kerberos.principal
.
SOLR_AUTHENTICATION_OPTS="\
...
-Dsolr.kerberos.cookie.domain=indigo20 \
-Dsolr.kerberos.principal=HTTP/indigo20@RED \
-Dsolr.kerberos.keytab=/opt/mr3-run/ranger/key/spnego.service.keytab"
solr-security.json
The user-role
section should specify service principals for HiveServer2 and Ranger.
"user-role": {
"hive/red0@RED": "updater",
"rangeradmin/indigo20@RED": "reader",
}
krb5.conf
This file should contains the information for Kerberos configuration.
Usually it suffices to use a copy of kubernetes/conf/krb5.conf
.
The user may have to comment out the variable renew_lifetime
if an error KrbApErrException: Message stream modified
occurs
(as explained in Accessing Secure HDFS).
Running Ranger
In order to run Ranger, the user can execute the script kubernetes/run-ranger.sh
:
$ kubernetes/run-ranger.sh
namespace/hivemr3 created
persistentvolume/workdir-pv-ranger created
persistentvolumeclaim/workdir-pvc-ranger created
configmap/hivemr3-ranger-conf-configmap created
secret/hivemr3-ranger-secret created
deployment/hivemr3-ranger created
service/ranger created
Executing the script kubernetes/run-ranger.sh
starts a Ranger Pod (consisting of two containers) in a moment:
$ kubectl get -n hivemr3 pods
NAME READY STATUS RESTARTS AGE
hivemr3-ranger-2q9wv 2/2 Running 0 33s
The user can check the log of the Ranger Pod to see if Ranger has successfully started:
$ kubectl logs -n hivemr3 hivemr3-ranger-2q9wv ranger
2020-10-09 07:19:37,935 --------- Running Ranger PolicyManager Web Application Install Script ---------
2020-10-09 07:19:37,942 [I] uname=Linux
2020-10-09 07:19:37,947 [I] hostname=hivemr3-ranger-2q9wv
2020-10-09 07:19:37,961 [I] DB_FLAVOR=MYSQL
...
Installation of Ranger PolicyManager Web Application is completed.
Starting Apache Ranger Admin Service
Apache Ranger Admin Service with pid 1661 has started.
Then the user can connect to the Ranger webpage at the address specified by the configuration key ranger.externalurl
.
The default ID/password is admin
/rangeradmin1
(where the password is given in kubernetes/ranger-key/install.properties
):
rangerAdmin_password=rangeradmin1
After starting Ranger, connect to the Ranger webpage and create a Ranger service specified in kubernetes/conf/ranger-hive-security.xml
.
Then fill the JDBC URL (e.g., jdbc:hive2://indigo20:9852/;principal=hive/red0@RED;
)
and set policy.download.auth.users
to the user hive
, or the owner of HiveServer2.
In this way, Ranger can inspect metadata (such as databases, tables, users) managed by HiveServer2 while HiveServer2 can retrieve its Ranger service profile.
Reconfiguring and running HiveServer2 after starting Ranger
After starting Ranger, HiveServer2 should be reconfigured so as to communicate with Ranger for checking data security.
If the Ranger plugin is missing in the directory kubernetes/hive/hive/apache-hive/lib
,
the user should rebuild the Docker image for running Hive on MR3 before restarting HiveServer2.
(The MR3 release already includes the Ranger plugin.)
kubernetes/yaml/hive.yaml
The spec/hostAliases
field should include the host running Ranger inside the Kubernetes cluster.
kubernetes/conf/hive-site.xml
The following configuration keys should be set:
<property>
<name>hive.security.authorization.manager</name>
<value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory</value>
</property>
<property>
<name>hive.security.authenticator.manager</name>
<value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>
</property>
The user may choose to set hive.server2.enable.doAs
to true because enabling impersonation is orthogonal to using Ranger.
kubernetes/conf/ranger-hive-security.xml
The configuration key ranger.plugin.hive.policy.rest.url
should use the host name assigned to Ranger.
Note that the port number should match the field port
, not targetPort
, in yaml/ranger-service.yaml
because HiveServer2 connects via the Service that exposes Ranger to the outside of the Kubernetes cluster.
<property>
<name>ranger.plugin.hive.policy.rest.url</name>
<value>http://indigo20:6080</value>
</property>
The configuration key ranger.plugin.hive.service.name
should use the Ranger service for HiveServer2:
<property>
<name>ranger.plugin.hive.service.name</name>
<value>INDIGO_hive</value>
</property>
kubernetes/conf/ranger-hive-audit.xml
The configuration key xasecure.audit.destination.solr.urls
should use the host name assigned to Ranger.
<property>
<name>xasecure.audit.destination.solr.urls</name>
<value>http://indigo20:6083/solr/ranger_audits</value>
</property>
Running Ranger without Kerberos
When running Ranger without Kerberos, SSL should not be enabled.
In order to run Ranger without Kerberos, the user should take the following additional steps.
kubernetes/ranger-conf/core-site.xml
Set the configuration key hadoop.security.authentication
to simple
to disable Kerberos authentication.
<property>
<name>hadoop.security.authentication</name>
<value>simple</value>
</property>
kubernetes/ranger-key/install.properties
Set the variable audit_solr_urls
to the address for the configuration key ranger.audit.solr.urls
.
audit_solr_urls=http://indigo20:6083/solr/ranger_audits
Remove (do not just set to empty) two variables related to authentication for auditing:
audit_solr_user
audit_solr_password
kubernetes/ranger-key/solr.in.sh
Set the following two variables:
SOLR_AUTH_TYPE="basic"
SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:solrRocks"
kubernetes/ranger-conf/solr-security.json
Set the configuration for authenticaion and authorization in Solr as follows:
{
"authentication": {
"blockUnknown": false,
"class": "solr.BasicAuthPlugin",
"credentials":{
"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="
}
},
"authorization": {
"class": "solr.RuleBasedAuthorizationPlugin"
}
}
Since authentication
/blockUnknown
is set to false,
Solr accepts audit requests without credentials.
(Ranger does not use the credentials which correspond to user solr
and password solrRocks
.)
Troubleshooting
1. HiveServer2 throws NullPointerException when trying to download Ranger policies, and Beeline cannot execute queries.
2020-10-08T12:23:08,872 ERROR [Thread-6] util.PolicyRefresher: PolicyRefresher(serviceName=ORANGE_hive): failed to refresh policies. Will continue to use last known version of policies (-1)
com.sun.jersey.api.client.ClientHandlerException: java.lang.RuntimeException: java.lang.NullPointerException
...
Caused by: java.lang.NullPointerException
0: jdbc:hive2://orange1:9852/> use tpcds_bin_partitioned_orc_1003;
Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [gitlab-runner] does not have [USE] privilege on [tpcds_bin_partitioned_orc_1003] (state=42000,code=40000)
The error disappears after setting policy.download.auth.users
to include the user of HiveServer2 in the Config Properties
panel.
2. Test Connection
fails in the Config Properties
panel.
Check if the jdbc.url
field is set properly.
Examples:
jdbc:hive2://indigo20:9852/
when neither Kerberos nor SSL is used.jdbc:hive2://indigo20:9852/;principal=hive/indigo20@RED;
when Kerberos is used.jdbc:hive2://indigo20:9852/;principal=hive/indigo20@RED;ssl=true;sslTrustStore=/opt/mr3-run/ranger/key/hivemr3-ssl-certificate.jks;
when both Kerberos and SSL are used.
3. Apache Ranger Admin Service fails to start.
In order to find out the cause of the failure, check out the file catalina.out
available in the Ranger container (not the Solr container) inside the Ranger Pod.
In the following example, Admin Service fails to start because a wrong Kerberos keytab file is provided.
$ kubectl -n hivemr3 logs hivemr3-ranger-cbq29 ranger
...
Installation of Ranger PolicyManager Web Application is completed.
Starting Apache Ranger Admin Service
Apache Ranger Admin Service failed to start!
$ kubectl exec -it -n hivemr3 hivemr3-ranger-cbq29 -c ranger /bin/bash;
root@hivemr3-ranger-cbq29:/opt/mr3-run/ranger# cat work-local-dir/log/ranger-admin/catalina.out
…
SEVERE: Tomcat Server failed to start:java.io.IOException: Login failure for rangeradmin/orange1@PL from keytab /opt/mr3-run/ranger/key/rangeradmin.keytab
java.io.IOException: Login failure for rangeradmin/orange1@PL from keytab /opt/mr3-run/ranger/key/rangeradmin.keytab
...
Caused by: java.security.GeneralSecurityException: Checksum failed
...