Spark-sql查詢正常

?

Hsql查詢異常

Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6

spark-shell查詢異常

查詢報錯

Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting

‘spark.debug.maxToStringFields’ in SparkEnv.conf.

無法獲取到數(shù)據(jù)

?

?Hsql查詢異常解決

參考鏈接：

hive2.1.1讀取spark寫入的orc:ORC split generation failed with exception:ArrayIndexOutOfBoundsException: 6_beetle_lzk的博客-CSDN博客

替換CDH集群的jar包(每個節(jié)點都需要替換，注意備份原文件)，重啟hive客戶端：

/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-2.1.1-cdh6.2.1.jar

/opt/cloudera/parcels/CDH/jars/hive-exec-2.1.1-cdh6.2.1.jar

/opt/cloudera/parcels/CDH/lib/spark/hive/hive-exec-2.1.1-cdh6.2.1.jar

spark-shell異常解決

參考鏈接：

Spark: "Truncated the string representation of a plan since it was too large." Warning when using manually created aggregation expression - Stack Overflow

[SPARK-15705] Spark won't read ORC schema from metastore for partitioned tables - ASF JIRA

spark-defaults.conf增加配置：

spark.debug.maxToStringFields=200

spark.sql.hive.convertMetastoreOrc=false

總結

以上是生活随笔為你收集整理的CDH6.2.1 hive2.1导入orc分区表异常处理过程的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內容還不錯，歡迎將生活随笔推薦給好友。