Ranger源碼編譯、使用手冊

1 Ranger簡介

Apache Ranger提供一個集中式安全管理框架,它可以對Hadoop生態的組件如Hive,Hbase進行細粒度的數據訪問控制.通過操作Ranger控制臺,管理員可以輕松的通過配置策略來控制用戶訪問HDFS文件夾、HDFS文件、數據庫、表、字段權限.這些策略可以為不同的用戶和組來設置,同時權限可與hadoop無縫對接.

2 準備

2.1 環境說明

1 Ranger源碼編譯依賴如下linux組件:maven,git,gcc,mysql

2 安裝git和gcc時采用yum安裝,請配置好本地源

3 安裝mysql是請確保清理好系統自帶的mysql.

2.1.1虛擬機里Linux系統版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

注:官網強調一定是1.7以上版本.

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14 ???|

+-----------+

注:

1 Mysql 驅動為mysql-connector-java-5.1.31-bin.jar

2 改jar被重命名后放置在/usr/share/java/內被其它Ranger插件共享

2.1.4 Maven版本

?[root@localhost bin]# mvn -version

Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;2014-02-15T01:37:52+08:00)

Maven home: /root/maven-3.2.1

Java version: 1.7.0_67,vendor: Oracle Corporation

Java home:/root/jdk1.7.0_67/jre

Default locale: en_US,platform encoding: UTF-8

OS name: "linux",version: "2.6.32-431.el6.x86_64", arch: "amd64", family:"unix"

2.1.5 git版本

?[root@localhost native]# git version

git version 1.7.1

2.2 編譯準備

2.2.1 安裝maven

[root@localhost ~]# cd /root

#下載地址

#https://maven.apache.org/download.cgi 最新版

#http://apache.opencas.org/maven/binaries/apache-maven-3.2.1-bin.tar.gz

tar –zxvf apache-maven-3.2.1-bin.tar.gz

mv apache-maven-3.2.1-bin maven-3.2.1

#修改環境變量,在~/.bash_profile里定義MAVEN_HOME并追加到PATH里

export MAVEN_HOME=/root/maven-3.2.1

:$MAVEN_HOME/bin:$PATH

#source環境變量,測試maven版本

source ~/.bash_profile

mvn –version

2.2.2 安裝git

這里通過本地源yum方式安裝.

yum install git

注:Linux本地源配置見下:

1) 虛擬機加載Linux ISO鏡像

2) 找到rom對應設備名

[root@localhost ~]# lsblk

NAME??????????????????????? MAJ:MIN RM? SIZE RO TYPE MOUNTPOINT

sr0????????????????????????? 11:0??? 1 1024M?0 rom?

sr1????????????????????????? 11:1??? 1?3.6G? 0 rom?

sr2????????????????????????? 11:2??? 1 1024M?0 rom?

sda?????????????????????????? 8:0??? 0???8G? 0 disk

?..sda1???????????????????????8:1??? 0?500M? 0 part /boot

?..sda2??????????????????????? 8:2??? 0?7.5G? 0 part

??..VolGroup-lv_root (dm-0) 253:0???0? 6.7G? 0 lvm?/

??..VolGroup-lv_swap (dm-1) 253:1???0? 816M? 0 lvm?[SWAP]

3) 創建目錄

[root@localhost ~]#mkdir –p /mnt/cdrom/

4) 掛載鏡像

mount -t iso9660 /dev/sr1?/mnt/cdrom

5) 配置linux更新源,/etc/yum.repos.d,修改成如下:

vi /etc/yum.repos.d/redhat.repo

[rhel-source]

name=Redhat

baseurl=file:///mnt/cdrom/

enabled=1

gpgcheck=1

gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release

6) 更新更新源

yum clean all

yum update list

2.2.3 安裝gcc

yum install gcc

2.2.4 安裝mysql

1) 安裝Mysql服務、客戶端

rpm –ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

2) 啟動mysql服務

service mysql start

3) 修改mysql初始密碼,先找到安裝時的初始密碼,在修改成自己的密碼

[root@localhost ~]#cat /root/.mysql_secret

# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY

mysql –uroot -p RUmKBqcY

set password=password(‘root’)

3 編譯

3.1編譯中

1) 拷貝ranger源代碼

[root@localhost ~]# cd ~

git clone https://github.com/apache/incubator-ranger.git

cd?incubator-ranger

git checkout ranger-0.5

2) 編譯ranger源代碼

cd?~/incubator-ranger

export?MAVEN_OPTS="-Xmx512M"?

export?JAVA_HOME= /root/jdk1.7.0_67

export?PATH=$JAVA_HOME/bin:$PATH

mvn clean compile package assembly:assembly?install

ls?target/*.tar.gz

[root@localhost ~]#ls/root/incubator-ranger/target/*.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hbase-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-kafka-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-kms.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-knox-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-migration-util.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-solr-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-src.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-storm-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-yarn-plugin.tar.gz

3.2 編譯問題

1) 編譯過程異常緩慢,一般要3-4天時間

2) 如果出現異常不好定位,可在maven參數里加-X以debug模式診斷

3) 源碼編譯的相關問題及解決方案

a) Failedto execute goal on project ranger-hdfs-plugin: Could not resolve dependenciesfor project

???? security_plugins.ranger-hdfs-plugin:

???? ranger-hdfs-plugin:jar:0.5.0:The following artifacts could not be resolved:

org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde,eigenbase:eigenbase-properties:jar:1.1.4,net.hydromatic:linq4j:jar:0.4,net.hydromatic:quidem:jar:0.1.1:

Could not transfer artifactorg.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde from/to conjars(http://conjars.org/repo): conjars.org:Unknownhost conjars.org -> [Help 1]

?

解決方案:手動下載錯誤提示里的jar并拷貝到相應的m2目錄內.

源:http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/

???? 目標:/root/.m2/repository/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde

? 源:http://conjars.org/repo/eigenbase/eigenbase-properties/1.1.4/

? 目標:/root/.m2/repository/eigenbase/eigenbase-properties/1.1.4/

? 源:http://conjars.org/repo/net/hydromatic/linq4j/0.4/

? 目標:/root/.m2/repository/net/hydromatic/linq4j/0.4/

? 源:http://conjars.org/repo/net/hydromatic/quidem/0.1.1/

? 目標:/root/.m2/repository/net/hydromatic/quidem/0.1.1/

?

b)[ERROR]error: error reading /root/.m2/repository/org/json/json/20090211/json-20090211.jar;zip file is empty

[ERROR] -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:Failed to execute goalorg.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) onproject ranger-hdfs-plugin: Compilation failure

error: error reading/root/.m2/repository/org/json/json/20090211/json-20090211.jar; zip file isempty

?

???? atorg.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)

.

解決方案:手動下載錯誤提示里的json-20090211.jar并拷貝到相應的m2目錄內.

c) Runningorg.apache.hadoop.crypto.key.kms.server.TestKeyAuthorizationKeyProvider此步耗時較長,大約20分鐘左右

4 配置

Ranger在solr里存儲日志,RangerAdmin UI依賴solr組件完成審計日志的查詢,所以需要先安裝和配置好Solr

注:目前(HDFS-Plugin)的測試日志審計時沒選擇Solr方式,但還是先配置好Standalone模式的solr.

4.1 Solr或SolrCloud安裝配置

cd /root/incubator-ranger/security-admin/contrib/solr_for_audit_setup

#打開install.properties文件,修改參數的值如下所示:

vi install.properties

JAVA_HOME=/root/jdk1.7.0_67

SOLR_INSTALL=true

SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz

SOLR_INSTALL_FOLDER=/opt/solr

SOLR_RANGER_HOME=/opt/solr/ranger_audit_server

SOLR_DEPLOYMENT=standalone

SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data

SOLR_LOG_FOLDER=/var/log/solr/ranger_audits

SOLR_MAX_MEM=2g

#安裝單節點的solr

./ setup.sh

#按照如下安裝提示啟動、停止solr服務

cat/opt/solr/ranger_audit_server/install_notes.txt

#啟動solr

/opt/solr/ranger_audit_server/scripts/start_solr.sh

#驗證solr服務是否可正常使用

lsof –i:6083

http://192.168.56.101:6083

#確保防火墻已經關閉

chkconfig?iptables?off

4.2 Ranger Admin安裝配置

cp/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz /root

cd /root

tar –zxvf ranger-0.5.0-admin.tar.gz

cd ranger-0.5.0-admin

#打開Ranger Admin里install.properties文件,修改參數的值如下所示:

vi install.properties

setup_mode=SeparateDB

DB_FLAVOR=MYSQL

db_root_user=root

db_root_password=root

db_host=localhost

db_name=ranger

db_user=root

db_password=root

audit_store=db

audit_db_name=ranger_audit

audit_db_user=root

audit_db_password=root

policymgr_external_url=http://localhost:6080

policymgr_http_enabled=true

unix_user=ranger

unix_group=ranger

#安裝Ranger Admin

./setup.sh

#啟動Ranger Admin服務

ranger-admin start

#驗證Ranger Admin服務,如果出現Ranger的登錄界面,說Okay了.注:用戶名/密碼 admin/admin

lsof –i:6080

http://192.168.56.101:6080

4.3 Ranger-usersync安裝配置

cp/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz /root/

cd /root

tar –zxvf ranger-0.5.0-usersync.tar.gz

cd ranger-0.5.0-usersync

#打開usersync Plugin里install.properties文件,修改參數的值如下所示:

vi install.properties

POLICY_MGR_URL=http://localhost:6080

SYNC_SOURCE=unix

#同步周期,1分鐘

SYNC_INTERVAL=1

logdir=/var/log/ranger/usersync

#安裝usersync Plugin

./setup.sh

#啟用usersync Plugin插件

./ranger--usersync-services.sh start

?

4.4 HDFS-Plugin安裝配置

cp/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz /root

cd /root

tar –zxvf ranger-0.5.0-hdfs-plugin.tar.gz

cd ranger-0.5.0-hdfs-plugin

#打開HDFS Plugin里install.properties文件,修改參數的值如下所示:

vi install.properties

POLICY_MGR_URL=http://localhost:6080

SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar

REPOSITORY_NAME=hadoopdev

XAAUDIT.DB.IS_ENABLED=true

XAAUDIT.DB.FLAVOUR=MYSQL

XAAUDIT.DB.HOSTNAME=localhost

XAAUDIT.DB.DATABASE_NAME=ranger_audit

XAAUDIT.DB.USER_NAME=root

XAAUDIT.DB.PASSWORD=root

#組件對應的用戶,這里設置為空.一般Hadoop的內置用戶是HDFS或則hadoop

CUSTOM_USER=root

CUSTOM_GROUP=root

#啟用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]#./enable-hdfs-plugin.sh

Customuser and group are not available, using default user and group.

ERROR:Unable to find the conf directory of component [hadoop]; dir [/root/hadoop/conf] not found.

Exitinginstallation.

注:這里報錯,需要額外將HADOOP的conf做個軟連接到/root/hadoop/conf.

ln-s /root/hadoop-2.7.1/etc/hadoop/ /root/hadoop/conf

#再次啟用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh

Custom user and group are not available,using default user and group.

ERROR: Unable to find the lib directory ofcomponent [hadoop];? dir [/root/hadoop/lib] not found.

Exiting installation.

#這里需要將HDFS Plugin內的jar和HADOOP包含的HDFS jar都指向/root/hadoop/lib

cp /root/ranger-0.5.0-hdfs-plugin/lib/ranger-hdfs-plugin-impl/*.jar/root/hadoop-2.7.1/share/hadoop/hdfs/lib/

mkdir /root/hadoop/lib

ln -s /root/hadoop-2.7.1/share/hadoop/hdfs/lib//root/hadoop/lib/

#再一次啟用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh

#驗證HDFS Plugin服務,這時登麗Ranger的管理員界面驗證下HDFS plugin是夠加載成功,發現并沒有.

?

原因是安裝HDFS plugin時install.properties文件里定義的REPOSITORY_NAME(值為hadoopdev)并未通過Ranger Admin在HDFS插件里的服務管理里注冊成服務(名hadoopdev).

解決方案:

1 登錄Ranger Adming

2 點擊HDFS plugin的添加按鈕

3 定義服務名為hadoopdev,提交其它信息后保存

#再次驗證HDFS plugin插件,則發現已經正常加載

注:

1 如果沒有安裝和開啟Ranger-usersync服務的情況下直接測試HDFS賦權權限是不成功的.

2 Ranger Admin的日志文件見 /root/ranger-0.5.0-admin/ews/logs/xa_portal.log

4.5 Hive-Plugin安裝配置

先啟動hive的metastore和hiveserver2服務

nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &

nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &

#beeline驗證

[root@localhost ~]# beeline -u"jdbc:hive2://192.168.56.101:10000" -n root -p test

Connectingto jdbc:hive2://192.168.56.101:10000

Connectedto: Apache Hive (version 1.2.1)

Driver:Hive JDBC (version 1.2.1)

Transactionisolation: TRANSACTION_REPEATABLE_READ

Beelineversion 1.2.1 by Apache Hive

0:jdbc:hive2://192.168.56.101:10000> show databases;

+----------------+--+

|database_name? |

+----------------+--+

|default??????? |

| shenl????????? |

+----------------+--+

?

2) Ranger-Admin里注冊hive plugin的服務

?

3) 配置、啟用hive plugin

cp/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz /root

cd /root

tar -zxvf ranger-0.5.0-hive-plugin.tar.gz

cd ranger-0.5.0-hive-plugin

#打開Hive Plugin里install.properties文件,修改參數的值如下所示:

vi install.properties

POLICY_MGR_URL=http://192.168.56.101:6080

REPOSITORY_NAME=hivedev

XAAUDIT.DB.IS_ENABLED=true

XAAUDIT.DB.FLAVOUR=MYSQL=MYSQL

XAAUDIT.DB.HOSTNAME=localhost

XAAUDIT.DB.DATABASE_NAME=ranger_audit

XAAUDIT.DB.USER_NAME=root

XAAUDIT.DB.PASSWORD=root

?

[root@localhostranger-0.5.0-hive-plugin]# ./enable-hive-plugin.sh

Customuser and group is available, using custom user and group.

ERROR:Unable to find the conf directory of component [hive]; dir [/root/hive/conf]not found.

Exitinginstallation.

#解決方法:

ln -s /root/apache-hive-0.13.0-bin/conf//root/hive/conf

?

[root@localhostranger-0.5.0-hive-plugin]# ./enable-hive-plugin.sh

Customuser and group is available, using custom user and group.

ERROR:Unable to find the lib directory of component [hive];? dir [/root/hive/lib] not found.

Exitinginstallation.

#解決方法:

cp root/ranger-0.5.0-hive-plugin/lib/ranger-hive-plugin-impl/*.jar/root/apache-hive-1.2.1-bin/lib/

mkdir /root/hive/lib

ln -s/root/apache-hive-1.2.1-bin/lib/ /root/hive/lib/

#將生成的hiveserver2-site.xml拷貝到hive的配置目錄下

cp /root/hive/conf/*/root/apache-hive-1.2.1-bin/conf/

#如果hive配置目錄里有hiveserver2-site.xml,則需要添加如下內容:

<property>

???????<name>hive.security.authorization.enabled</name>

??????? <value>true</value>

??? </property>

??? <property>

???????<name>hive.security.authorization.manager</name>

??????? <value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory</value>

??? </property>

??? <property>

???????<name>hive.security.authenticator.manager</name>

???????<value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>

??? </property>

??? <property>

???????<name>hive.conf.restricted.list</name>

???????<value>hive.security.authorization.enabled,hive.security.authorization.manager,hive.security.authenticator.manager</value>

??? </property>

5 使用

5.1 HDFS Plugin驗證

1 新增linux用戶shenl進行測試

useradd shenl

hadoop fs –mkdir /shenl

2 登錄Ranger Admin,新增用戶shenl

3 登錄Ranger Admin,在hadoopdev里注冊新的Policy

4 添加該Policy的權限,指定可以對HDFS里的/shenl有R權限

#切換到shenl用戶進行put權限測試,應該報錯.

5 編輯Policy追加shenl用戶的W權限

#此時shenl用戶應該可以擁有/shenl目錄的put權限

5.2 Hive Plugin驗證

1 hive plugin加載驗證

2 定義權限策略

3 beeline里權限驗證

6 總結

無 與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的Apache Ranger源码编译及使用的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。