sqoop2操作流程
--------------------------配置-------------------------------------
配置/home/appleyuchi/bigdata/sqoop-1.99.5-bin-hadoop200/server/conf
中的sqoop.properties文件以及catalina.properties文件
catalina.properties中修改如下:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/common/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/common/lib/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/hdfs/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/hdfs/lib/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/mapreduce/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/mapreduce/lib/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/yarn/*.jar,/home/appleyuchi/bigdata/hadoop-3.0.3/share/hadoop/yarn/lib/*.jar
sqoop.properties中修改如下:
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/appleyuchi/bigdata/hadoop-3.0.3/etc/hadoop
--------------------------配置檢查-------------------------------------
先運行./sqoop2-tool verify
檢查配置,如果報告fail
那么檢查下面的log看看是哪里的問題
/home/appleyuchi/bigdata/sqoop-1.99.5-bin-hadoop200/bin/@LOGDIR@/sqoop.log
---------------------------接來下是啟動-------------------------------------
啟動分為兩步,先啟動服務,然后再啟動客戶端:
1.
(python2.7) appleyuchi@ubuntu:~/bigdata/sqoop-1.99.5-bin-hadoop200/bin$ sqoop2-server start
2.
(python2.7) appleyuchi@ubuntu:~/bigdata/sqoop-1.99.5-bin-hadoop200/bin$ sqoop2-shell
---------------------------接來下是常見基本操作---------------------------------
set server --host 127.0.0.1 --port 12000 --webapp sqoop
sqoop:000> 模式下可以執行以下命令:
show version
show version --all(如果這里報告有exception,那么說明前面的配置沒有成功)
sqoop:000> show connector
+----+------------------------+---------+------------------------------------------------------+----------------------+
| Id | ? ? ? ? ?Name ? ? ? ? ?| Version | ? ? ? ? ? ? ? ? ? ? ? ?Class ? ? ? ? ? ? ? ? ? ? ? ? | Supported Directions |
+----+------------------------+---------+------------------------------------------------------+----------------------+
| 1 ?| generic-jdbc-connector | 1.99.5 ?| org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO ? ? ? ? ? ? ?|
| 2 ?| kite-connector ? ? ? ? | 1.99.5 ?| org.apache.sqoop.connector.kite.KiteConnector ? ? ? ?| FROM/TO ? ? ? ? ? ? ?|
| 3 ?| hdfs-connector ? ? ? ? | 1.99.5 ?| org.apache.sqoop.connector.hdfs.HdfsConnector ? ? ? ?| FROM/TO ? ? ? ? ? ? ?|
| 4 ?| kafka-connector ? ? ? ?| 1.99.5 ?| org.apache.sqoop.connector.kafka.KafkaConnector ? ? ?| TO ? ? ? ? ? ? ? ? ? |
+----+------------------------+---------+------------------------------------------------------+----------------------+
上面的這個到底什么作用呢?
我們可以看到左側的id有1,2,3,4,這個其實就是后面的cid,所以cid不能亂用,你想用sqoop2處理什么樣的數據庫,就要使用什么樣的cid
所謂的cid就是connector的id
sqoop:000> show link
+----+-----------+-----------+---------+
| Id | ? Name ? ?| Connector | Enabled |
+----+-----------+-----------+---------+
| 1 ?| hdfs_link | 3 ? ? ? ? | true ? ?|
+----+-----------+-----------+---------+
sqoop:000> delete link --lid 1
sqoop:000> show link
+----+------+-----------+---------+
| Id | Name | Connector | Enabled |
+----+------+-----------+---------+
+----+------+-----------+---------+
sqoop:000> show job
+----+------+----------------+--------------+---------+
| Id | Name | From Connector | To Connector | Enabled |
+----+------+----------------+--------------+---------+
+----+------+----------------+--------------+---------+
------------------------------下面是正式操作--------------------------------------------------------------------------
主要內容分為兩部分:
一:
mysql導入hive
二:
hive導入mysql
☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆
sqoop:000> set server --host 127.0.0.1 --port 12000 --webapp sqoop
☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆先創建第一個link☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆
sqoop:000> create link --cid 1 ? ? ? ? ? ? ? ? ??
注意上面的cid不是隨便寫的,是根據上面的
sqoop:000> show connector命令的返回結果得到的表格中第一列id查詢得到的,
想用什么數據庫,就在此處cid的后面使用表格中對應驅動的id
Creating link for connector with id 1
Please fill following values to create new link object
Name: mysql
Link configuration
JDBC Driver Class:com.mysql.jdbc.Driver?
JDBC Connection String: jdbc:mysql://127.0.0.1:3306/employees
Username: root
Password: **********
JDBC Connection Properties:?
There are currently 0 values in the map:
entry#?
New link was successfully created with validation status OK and persistent id 2
sqoop:000> show link
+----+-------+-----------+---------+
| Id | Name ?| Connector | Enabled |
+----+-------+-----------+---------+
| 2 ?| mysql | 1 ? ? ? ? | true ? ?|
+----+-------+-----------+---------+
☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆再創建第2個link☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆
sqoop:000> create link --cid 3
Creating link for connector with id 3
Please fill following values to create new link object
Name: hdfs
Link configuration
HDFS URI: hdfs://localhost:9000/user/appleyuchi/test.txt
New link was successfully created with validation status OK and persistent id 3
sqoop:000> show link
+----+-------+-----------+---------+
| Id | Name ?| Connector | Enabled |
+----+-------+-----------+---------+
| 2 ?| mysql | 1 ? ? ? ? | true ? ?|
| 3 ?| hdfs ?| 3 ? ? ? ? | true ? ?|
+----+-------+-----------+---------+
☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆注意連接關系:數據庫-驅動-link-job-link-驅動-數據庫☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆
sqoop:000> create job -f 2 -t 3
后面的選擇中,必填項有:
Schema name: employees
Table name: departments
Choose: 0
Choose: 0
Output directory: ~jdbc2hdfs?
其他一律回車鍵跳過即可
☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆啟動job☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆
報錯:
sqoop:000> start job --jid 1
Exception has occurred during processing command?
Exception: java.lang.RuntimeException Message: java.lang.ClassNotFoundException: org.apache.sqoop.driver.DriverError
這個問題網上查不到,百度和google都不行,只能放棄了...
最后只好把sqoop2給卸載了.
?
根據
https://stackoverflow.com/questions/41388979/what-does-sqoop-2-provide-that-sqoop-1-does-not
可知：
目前發展趨勢是使用sqoop1,sqoop2面臨淘汰。
------------------------------------附錄----------------------------------------------------------------
二、hdfs資源uri格式：
用法：scheme://authority/path
選項：
scheme–>協議名，file或hdfs
authority–>namenode主機名
path–>路徑
范例：hdfs://localhost:54310/user/hadoop/test.txt
假設已經在/home/hadoop/hadoop-1.1.1/conf/core-site.xml里配置了fs.default.name=hdfs://localhost:54310，則僅使用/user/hadoop/test.txt即可。hdfs默認工作目錄為/user/$USER，$USER是當前的登錄用戶名。
總結
以上是生活随笔為你收集整理的sqoop2操作流程的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 安装sqoop以后,hive中的info
- 下一篇： 正确地启动hadoop