1。引發：做一個java讀取hbase的注冊成表的程序。但是讀出來的是javaPairRDD，而網上都是javaRDD轉成dataFrame,我只能自己摸索怎么轉成javaRDD?
2。方法?
　　JavaRDD => JavaPairRDD: 通過mapToPair函數?
　　JavaPairRDD => JavaRDD: 通過map函數轉換?
3。不管其他先運行一下

package com.lcc.spark.rdd.test;import java.util.ArrayList; import java.util.Arrays; import java.util.List;import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.Function; import org.apache.spark.api.java.function.Function2; import org.apache.spark.api.java.function.PairFunction; import org.apache.spark.api.java.function.VoidFunction; import org.apache.spark.storage.StorageLevel;import com.google.common.base.Optional;import scala.Tuple2; import shapeless.Tuple; /** * Hello world! * */ public class App { public static void main(String[] args) { SparkConf conf = new SparkConf().setMaster("local").setAppName("Simple Application"); JavaSparkContext sc = new JavaSparkContext(conf); // convert from other RDD JavaRDD<String> line1 = sc.parallelize(Arrays.asList("1 aa", "2 bb", "4 cc", "3 dd")); line1.foreach(new VoidFunction<String>(){@Overridepublic void call(String num) throws Exception {// TODO Auto-generated method stubSystem.out.println("numbers;"+num);}});JavaPairRDD<String, String> prdd = line1.mapToPair(new PairFunction<String, String, String>() { public Tuple2<String, String> call(String x) throws Exception { return new Tuple2(x.split(" ")[0], x.split(" ")[1]); } }); System.out.println("111111111111mapToPair:"); prdd.foreach(new VoidFunction<Tuple2<String, String>>() { public void call(Tuple2<String, String> x) throws Exception { System.out.println(x); } }); /* JavaRDD => JavaPairRDD: 通過mapToPair函數JavaPairRDD => JavaRDD: 通過map函數轉換*/System.out.println("===============1========="); JavaRDD<String> javaprdd =prdd.map(new Function<Tuple2<String,String>,String>() {private static final long serialVersionUID = 1L;@Overridepublic String call(Tuple2<String, String> arg0) {// TODO Auto-generated method stubSystem.out.println("arg0======================"+arg0);System.out.println("arg0======================"+arg0._1);return arg0._1+" "+arg0._2;}});System.out.println("===============2========="); javaprdd.foreach(new VoidFunction<String>(){@Overridepublic void call(String num) throws Exception {// TODO Auto-generated method stubSystem.out.println("numbers;"+num);}});} }

結果如下

numbers;1 aa numbers;2 bb numbers;4 cc numbers;3 dd111111111111mapToPair: (1,aa) (2,bb) (4,cc) (3,dd)===============1========= ===============2========= arg0======================(1,aa) arg0======================1 numbers;1 aa arg0======================(2,bb) arg0======================2 numbers;2 bb arg0======================(4,cc) arg0======================4 numbers;4 cc arg0======================(3,dd) arg0======================3 numbers;3 dd

4。分析

JavaPairRDD<String, String> prdd = line1.mapToPair(new PairFunction<要返回JavaPairRDD的key類型, 要返回JavaPairRDD的value類型, javaRDD要輸入的類型>() { public Tuple2<要返回JavaPairRDD的key類型, 要返回JavaPairRDD的value類型> call(javaRDD要輸入的類型 x) throws Exception { return new Tuple2(x.split(" ")[0], x.split(" ")[1]); } }); JavaPairRDD<String, String> prdd = line1.mapToPair(new PairFunction<String, String, String>() { public Tuple2<String, String> call(String x) throws Exception { return new Tuple2(x.split(" ")[0], x.split(" ")[1]); } }); JavaRDD<要返回JavaRDD的類型> javaprdd =prdd.map(new Function<Tuple2<JavaPairRDD的key類型,JavaPairRDD的value類型>,要返回JavaRDD的類型>() {private static final long serialVersionUID = 1L;@Overridepublic String call(Tuple2<String, String> arg0) {// TODO Auto-generated method stubSystem.out.println("arg0======================"+arg0);System.out.println("arg0======================"+arg0._1);return arg0._1+" "+arg0._2;}});System.out.println("===============1========="); JavaRDD<String> javaprdd =prdd.map(new Function<Tuple2<String,String>,String>() {private static final long serialVersionUID = 1L;@Overridepublic String call(Tuple2<String, String> arg0) {// TODO Auto-generated method stubSystem.out.println("arg0======================"+arg0);System.out.println("arg0======================"+arg0._1);return arg0._1+" "+arg0._2;}});System.out.println("===============2=========");

從輸出結果看

===============1========= ===============2========= 比下面的map函數體的內容先輸出，我在這里坑了很久， 明明map函數寫對了，但是我卻沒在下面打印輸出， 結果總是map函數體的沒有執行，我就很納悶啊， 為什么啊，為什么不對啊，也沒報錯啊， 但是我一打印才發現里面竟然執行了， 這時候我突然想起spark的程序都是懶加載的， 你用的時候他才會執行，所以上面兩行會先打印出來。 都是淚啊 arg0======================(1,aa) arg0======================1 numbers;1 aa

5。由此總結javaRDD怎么轉換成JavaPairRDD的就，反著怎么轉換回去，互為逆運算

總結

以上是生活随笔為你收集整理的spark学习：java版JavaRDD与JavaPairRDD的互相转换的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。