Apache Sentry手动安装、使用手册
Sentry手動安裝、使用手冊
1 Sentry簡介
Apache Sentry 是Cloudera公司發布的一個Hadoop開源組件，截止目前還是Apache的孵化項目，它提供了細粒度級、基于角色的授權以及多租戶的管理模式。Sentry當前可以和Hive/Hcatalog、Apache Solr 和Cloudera Impala集成，未來會擴展到其他的Hadoop組件，例如HDFS和HBase.
2 準備
2.1 環境說明
1 Sentry安裝采用rpm包的方式.
2 Hadoop版本為hadoop-2.5.0-cdh5.3.3,Hive版本為hive-0.13.1-cdh5.3.3,Sentry版本為sentry-1.4.0-cdh5.3.3
3 Sentry下載地址:
http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.3.3/RPMS/noarch/
2.1.1虛擬機里Linux系統版本
[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux
Red Hat Enterprise Linux Server release 6.5 (Santiago)
2.1.2 JDK版本
[root@localhost native]# java -version
java version "1.7.0_67"
Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)
2.1.3mysql版本
[root@localhost native]# mysql -uroot -proot-e"select version()";
Warning: Using a password onthe command line interface can be insecure.
+-----------+
| version() |
+-----------+
| 5.6.14 ???|
+-----------+
注:
1 Mysql 驅動為mysql-connector-java-5.1.31-bin.jar
2 改jar被重命名后放置在/usr/share/java/內被其它Ranger插件共享
3 安裝
3.1安裝Mysql
1)安裝mysql相關服務
rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm
rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm時報如下錯誤:
file/usr/share/mysql/charsets/macroman.xml from install of MySQL-server-5.6.14-1.el6.x86_64conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64 file/usr/share/mysql/charsets/swe7.xml from install ofMySQL-server-5.6.14-1.el6.x86_64 conflicts with file from packagemysql-libs-5.1.71-1.el6.x86_64
rpm -q mysql-libs-5.1.71-1.el6.x86_64
rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm
ARANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
Youwill find that password in '/root/.mysql_secret'.(生成mysql root密碼)
Youmust change that password on your first connect,
noother statement but 'SET PASSWORD' will be accepted.
rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm
service mysql start
#這里密碼 來自/root/.mysql_secret
mysql -uroot -p9RNrbk9O
?
#首次執行會提示修改mysql root密碼
SET PASSWORD=PASSWORD('root');
#創建Hive數據庫為Hive的元數據庫
create database hive;
GRANT all ON hive.* TO root@'%' IDENTIFIED BY 'root';
#創建sentry數據庫為sentry元數據庫
create database sentry;
CREATE USER sentry IDENTIFIED BY 'sentry';
GRANT all ON sentry.* TO sentry@'%' IDENTIFIED BY'sentry';
flush privileges;
?
3.2 安裝Hive
1)解壓Hive壓縮包,并配置環境變量
cd /root
tar –zxvf hive-0.13.1-cdh5.3.3.tar.gz
vi ~/.bash_profile
exportHIVE_HOME=/root/hive-0.13.1-cdh5.3.3
#追加HIVE執行文件路徑到PATH內
exportPATH=$PATH:$HIVE_HOME/bin
2) 拷貝Mysql驅動到Hive的lib目錄下:
cp???? /root/mysql-connector-java-5.1.31-bin.jar/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar
?
3)配置Hive的conf.詳細的hive-site.xml配置內容見下:
<?xmlversion="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <configuration> <property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value><description>JDBC connect string for aJDBC metastore</description> </property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value><description>Driver class name for aJDBC metastore</description> </property><property><name>javax.jdo.PersistenceManagerFactoryClass</name><value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value><description>class implementing the jdopersistence</description> </property><property><name>javax.jdo.option.DetachAllOnCommit</name><value>true</value><description>detaches all objects fromsession so that they can be used after transaction iscommitted</description> </property><property><name>javax.jdo.option.NonTransactionalRead</name><value>true</value><description>reads outside oftransactions</description> </property><property><name>javax.jdo.option.ConnectionUserName</name><value>root</value><description>username to use againstmetastore database</description> </property><property><name>javax.jdo.option.ConnectionPassword</name><value>root</value><description>password to use againstmetastore database</description> </property><property><name>javax.jdo.option.Multithreaded</name><value>true</value><description>Set this to true ifmultiple threads access metastore through JDO concurrently.</description> </property></configuration>3.3 安裝Sentry
1)安裝Sentry相關服務
rpm -ivh --nodepssentry-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
rpm -ivh --nodeps sentry-hdfs-plugin-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
rpm -ivh --nodepssentry-store-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
2)替換Sentry內的Hadoop、Hive、Impala、Hbase、Zookeeper、Parquet、Avro等jar包
rm -rf /usr/lib/sentry/lib/hive*.jar
rm -rf /usr/lib/sentry/lib/hadoop*.jar
rm -rf /usr/lib/sentry/lib/zookeeper*.jar
rm -rf /usr/lib/sentry/lib/avro*.jar
rm -rf /usr/lib/sentry/lib/server/hive-beeline.jar
cp ~/SentryLibs/* /usr/lib/sentry/lib/
注:這里的Sentrylibs里的jar包是通過/usr/lib/sentry/lib里的文件名分別在Hadoop、Hive、Hbase等里的安裝目錄里一一找到的.
mv /root/SentryLibs/hive-beeline.jar/usr/lib/sentry/lib/server/??
#拷貝mysql驅動到sentry的lib目錄內?
cp/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar/usr/lib/sentry/lib/
#如果裝了Impala,此步可忽略
rpm -ivh bigtop-utils-0.7.0+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6.noarch.rpm
?
3) 配置Sentry,詳細配置見sentry-site.xml內容如下:
<?xmlversion="1.0" encoding="UTF-8"?> <configuration><property><name>sentry.service.security.mode</name><value>none</value></property><property><name>sentry.service.admin.group</name><value>impala,hive</value></property><property><name>sentry.service.allow.connect</name><value>impala,hive</value></property><property><name>sentry.verify.schema.version</name><value>true</value></property><property><name>sentry.service.server.rpc-address</name><value>data1</value></property><property><name>sentry.service.server.rpc-port</name><value>8038</value></property><property><name>sentry.store.jdbc.url</name><value>jdbc:mysql://localhost:3306/sentry</value></property><property><name>sentry.store.jdbc.driver</name><value>com.mysql.jdbc.Driver</value></property><property><name>sentry.store.jdbc.user</name><value>sentry</value></property><property><name>sentry.store.jdbc.password</name><value>sentry</value></property><property><name>sentry.hive.server</name><value>data1</value></property><property><name>sentry.store.group.mapping</name><value>org.apache.sentry.provider.common.HadoopGroupMappingService</value></property> </configuration>4)初始化Sentry元數據
sentry --command schema-tool --conffile/etc/sentry/conf/sentry-site.xml --dbType mysql --initSchema
??????????????????..........................
??????????????????No rows affected (0.094 seconds)
??????????????????No rows affected (0.015 seconds)
??????????????????No rows affected (0.075 seconds)
??????????????????1 row affected (0.007 seconds)
??????????????????Closing: 0: jdbc:mysql://localhost:3306/sentry
??????????????????Initialization script completed
??????????????????Sentry schemaTool completed
?
3.4 安裝問題
暫無.
4 配置
Ranger在solr里存儲日志,RangerAdmin UI依賴solr組件完成審計日志的查詢,所以需要先安裝和配置好Solr
注:目前(HDFS-Plugin)的測試日志審計時沒選擇Solr方式,但還是先配置好Standalone模式的solr.
4.1 Hive集成Sentry配置
1) 拷貝Sentry相關jar包到Hive的lib目錄內
?cp /usr/lib/sentry/lib/sentry*.jar$HIVE_HOME/lib/
?cp /usr/lib/sentry/lib/shiro-core-*.jar$HIVE_HOME/lib/
2)? Hive conf內新增Sentry-site.xml,配置內容見下:
<?xmlversion="1.0" encoding="UTF-8"?> <configuration><property><name>sentry.service.security.mode</name><value>none</value> </property> <property><name>sentry.service.server.principal</name><value>hivemeta/centos64.cloudera.com@HS2.CLOUDERA.COM</value> </property><property><name>sentry.service.client.server.rpc-port</name><value>8038</value></property><property><name>sentry.service.client.server.rpc-address</name><value>data1</value></property><property><name>sentry.service.client.server.rpc-connection-timeout</name><value>200000</value></property><property><name>sentry.hive.provider</name><value>org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider</value></property><property><name>sentry.hive.provider.backend</name><value>org.apache.sentry.provider.db.SimpleDBProviderBackend</value></property><property><name>sentry.hive.server</name><value>server1</value></property><property><name>sentry.metastore.service.users</name><value>root</value></property><property><name>sentry.hive.testing.mode</name><value>true</value></property> </configuration>3)? 修改Hive conf內的hive-site.xml,修改成如下:
<?xmlversion="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <configuration> <!--########################################################################################--> <!--################################ sentry for metastore############################################# --> <!-- <property><name>hive.metastore.client.impl</name><value>org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient</value><description>Sets custom Hivemetastore client which Sentry uses to filter out metadata.</description> </property> --> <property><name>hive.metastore.execute.setugi</name><value>true</value><description>In unsecure mode, setting this propertyto true will cause the metastore to execute DFS operations usingthe client's reported user and grouppermissions. Note that this property must be set onboth the client and server sides. Furthernote that its best effort.If client sets its to true and serversets it to false, client setting will be ignored.</description></property> <property><name>hive.metastore.pre.event.listeners</name><value>org.apache.sentry.binding.metastore.MetastoreAuthzBinding</value> </property><property><name>hive.metastore.event.listeners</name><value>org.apache.sentry.binding.metastore.SentryMetastorePostEventListener</value> </property><property><name>hive.server2.enable.impersonation</name><value>true</value> </property> <property><name>hive.security.authorization.task.factory</name><value>org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl</value> </property> <property><name>hive.server2.session.hook</name><value>org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook</value> </property> <property><name>hive.sentry.conf.url</name><value>file:///root/hive-0.13.1-cdh5.3.3/conf/sentry-site.xml</value> </property> <!-- <property><name>hive.metastore.uris</name><value>thrift://data1:9083</value><description>Thrift URI for the remotemetastore. Used by metastore client to connect to remotemetastore.</description> </property> --> <property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value><description>JDBC connect string for aJDBC metastore</description> </property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value><description>Driver class name for aJDBC metastore</description> </property> <property><name>javax.jdo.PersistenceManagerFactoryClass</name><value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value><description>class implementing the jdopersistence</description> </property> <property><name>javax.jdo.option.DetachAllOnCommit</name><value>true</value><description>detaches all objects fromsession so that they can be used after transaction iscommitted</description> </property> <property><name>javax.jdo.option.NonTransactionalRead</name><value>true</value><description>reads outside of transactions</description> </property> <property><name>javax.jdo.option.ConnectionUserName</name><value>root</value><description>username to use againstmetastore database</description> </property> <property><name>javax.jdo.option.ConnectionPassword</name><value>root</value><description>password to use againstmetastore database</description> </property> <property><name>javax.jdo.option.Multithreaded</name><value>true</value><description>Set this to true ifmultiple threads access metastore through JDO concurrently.</description> </property> </configuration>4)? 驗證Sentry的權限控制,詳見 5 使用:
4.2 Impala集成Sentry配置
1)? Impala的conf里新增sentry配置,/etc/impala/conf/sentry-site.xml內容如下:
<?xmlversion="1.0" encoding="UTF-8"?> <configuration> <property><name>sentry.service.client.server.rpc-port</name><value>8038</value> </property> <property><name>sentry.service.client.server.rpc-address</name><value>data1</value> </property> <property><name>sentry.service.client.server.rpc-connection-timeout</name><value>200000</value> </property> <property><name>sentry.service.security.mode</name><value>none</value> </property> </configuration>2)? 修改impala的參數
vi/etc/default/impala
IMPALA_CATALOG_ARGS 參數里追加 -sentry_config=/etc/impala/conf/sentry-site.xml
IMPALA_SERVER_ARGS 參數里追加 -server_name=server1\
???-sentry_config=/etc/impala/conf/sentry-site.xml
5 使用
5.1 Hive集成Sentry權限驗證
1) 啟動sentry服務,hive的metastore和hiveserver2服務
sentry --command service--conffile /etc/sentry/conf/sentry-site.xml
nohup?hive --service metastore-hiveconf hive.root.logger=INFO,console?> myout1.file 2>&1 &
nohup?hiveserver2 -hiveconfhive.root.logger=INFO,console?> myout2.file 2>&1 &
#hiveserver2里新增hive和test用戶,通過beeline方式訪問.
groupadd hive; useradd hive -ghive;passwd hive (type hive123)
groupadd test; useradd test -gtest;passwd test (type test123)
beeline -u "jdbc:hive2://data1:10000"-n hive -p test
?
2)驗證整體思路及場景概述
思路:
1 HIVE用戶屬于管理員組,服務整個server的權限
2 Test用戶創建兩個數據庫test_only(all權限),test_select_only(僅有select權限)
具體驗證:
1 通過show databases和use database驗證訪問權限
2 通過test用戶的use test_only能drop table,create table驗證all權限
3 通過test用戶的use test_select_only不能drop table驗證僅有select權限
#場景一 root用戶連入,嘗試執行DDL操作.(無權操作)
[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n root
??????????????????scan complete in 17ms
??????????????????Connecting to jdbc:hive2://data1:10000
??????????????????Connected to: Apache Hive (version 0.13.1-cdh5.3.3)
??????????????????Driver: Hive JDBC (version 0.13.1-cdh5.3.3)
??????????????????Transaction isolation: TRANSACTION_REPEATABLE_READ
??????????????????Beeline version 0.13.1-cdh5.3.3 by Apache Hive
0:jdbc:hive2://data1:10000>create database sensitive;
????????Error: Error while compiling statement: FAILED:SemanticException No valid privileges
????????Required privileges for this query: Server=server1->action=*;(state=42000,code=40000)
????????0: jdbc:hive2://data1:10000> create role admin_role;
????????Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.SentryAccessDeniedException: Access denied to root (state=08S01,code=1)
0:jdbc:hive2://data1:10000> !q
????????Closing: 0: jdbc:hive2://data1:10000
#場景二 hive用戶連入,對用戶hive、test進行授權.
[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n hive
??????????????????scan complete in 5ms
??????????????????Connecting to jdbc:hive2://data1:10000
??????????????????Connected to: Apache Hive (version 0.13.1-cdh5.3.3)
??????????????????Driver: Hive JDBC (version 0.13.1-cdh5.3.3)
??????????????????Transaction isolation: TRANSACTION_REPEATABLE_READ
??????????????????Beeline version 0.13.1-cdh5.3.3 by Apache Hive
0:jdbc:hive2://data1:10000>create role admin_role;
????????No rows affected (1.383 seconds)
GRANT ALL ONSERVER server1 TO ROLE admin_role;
GRANT ROLEadmin_role TO GROUP hive;
create roletest_role;
GRANT ALL ONDATABASE test_only TO ROLE test_role;
GRANT ROLEtest_role TO GROUP test;
GRANT SELECT ONDATABASE test_select_only TO ROLE test_role;
#場景三 test用戶連入,對數據庫admin_only、test_select_only進行權限驗證.
beeline -u"jdbc:hive2://data1:10000" -n test
0:jdbc:hive2://data1:10000> showdatabases;
??????????????????+----------------+--+
??????????????????| database_name? |
??????????????????+----------------+--+
??????????????????| admin_only???? |
??????????????????| default??????? |
??????????????????| test_only????? |
??????????????????+----------------+--+
??????????????????3 rows selected (0.721 seconds)
0:jdbc:hive2://data1:10000>use admin_only;
??????????????????Error: Error while compiling statement: FAILED:SemanticException No valid privileges
??????????????????Required privileges for this query:Server=server1->Db=admin_only->Table=*->action=insert;Server=server1->Db=admin_only->Table=*->action=select;(state=42000,code=40000)???????????
0:jdbc:hive2://data1:10000>use test_select_only;
??????????????????No rows affected (0.313 seconds)
0:jdbc:hive2://data1:10000>show tables;
??????????????????+--------------+--+
??????????????????|?? tab_name?? |
??????????????????+--------------+--+
??????????????????| select_only? |
??????????????????+--------------+--+
??????????????????1 row selected (0.337 seconds)
0:jdbc:hive2://data1:10000>drop table select_only;
??????????????????Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.DDLTask.MetaException(message:hive.metastore.execute.setugi can't be false in nonsecure mode) (state=08S01,code=1)
??? #這里需要在HIVE服務端添加如下參數
??????????????????<property>
???????????????????????????<name>hive.metastore.execute.setugi</name>
???????????????????????????<value>true</value>
??????????????????</property>
??????????????????
??????????????????4 rows selected (2.9 seconds)
0:jdbc:hive2://data1:10000>use test_select_only;
?????????????????? Norows affected (0.46 seconds)
0:jdbc:hive2://data1:10000>show tables;
??????????????????+--------------+--+
??????????????????|?? tab_name?? |
??????????????????+--------------+--+
??????????????????| select_only? |
??????????????????+--------------+--+
??????????????????1 row selected (0.507 seconds)
0:jdbc:hive2://data1:10000>drop table select_only;
??????????????????Error: Error while compilingstatement: FAILED: SemanticException No valid privileges
??????????????????Required privileges for this query:Server=server1->Db=test_select_only->Table=select_only->action=*;(state=42000,code=40000)
#場景四 test用戶連入,對數據庫test_only進行驗證,擁有所有權限:可以查看、刪除表
0: jdbc:hive2://data1:10000>usetest_only;
??????????????????No rows affected (0.819 seconds)
0:jdbc:hive2://data1:10000> showtables;
??????????????????+--------------+--+
??????????????????|?? tab_name?? |
??????????????????+--------------+--+
??????????????????| test_itself? |
??????????????????+--------------+--+
??????????????????1 row selected (0.426 seconds)
0:jdbc:hive2://data1:10000>drop table test_itself;
??????????????????No rows affected (6.336 seconds)
??????????????????
0: jdbc:hive2://data1:10000> create table test_newone ( ont string);
?????????????????? Norows affected (1.377 seconds)
0:jdbc:hive2://data1:10000>show tables;
??????????????????+--------------+--+
??????????????????|?? tab_name?? |
??????????????????+--------------+--+
??????????????????| test_newone? |
??????????????????+--------------+--+
5.2 Impala集成Sentry權限驗證
1)? 進入impala-shell,執行權限驗證
create role supervisor;
grant all on server to rolesupervisor;
grant role supervisor to groupimpala;
GRANT ALL ON SERVER server1 TOROLE any_operation;
GRANT ROLE any_operation TOGROUP hive;
grant ALL ON database EDA TOtest_role;
grant select ON databasepriselect TO test_role;
2)? 切換至test用戶測試impala權限:
?
6 總結
仔細跟蹤log,具體問題具體分析.
總結
以上是生活随笔為你收集整理的Apache Sentry手动安装、使用手册的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 宏?笔记本bios怎么降级 宏碁笔记本如
- 下一篇： 哪些旅游城市房产适合外地人投资