Spark1.4 和 Hive 1.1.1 启动错误
生活随笔
收集整理的這篇文章主要介紹了
Spark1.4 和 Hive 1.1.1 启动错误
小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.
啟動(dòng)Spark-sql
啟動(dòng)Spark-shell [jifeng@feng03 hive]$ ./bin/spark-shell --master spark://feng03:7077 --driver-class-path /home/jifeng/hive/mysql-connector-java-5.1.34.jar -bash: ./bin/spark-shell: No such file or directory [jifeng@feng03 hive]$ cd .. [jifeng@feng03 ~]$ cd spark-1.4.0-bin-hadoop2.6/ [jifeng@feng03 spark-1.4.0-bin-hadoop2.6]$ ./bin/spark-shell --master spark://feng03:7077 --driver-class-path /home/jifeng/hive/mysql-connector-java-5.1.34.jar 15/09/05 14:50:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/09/05 14:50:29 INFO spark.SecurityManager: Changing view acls to: jifeng 15/09/05 14:50:29 INFO spark.SecurityManager: Changing modify acls to: jifeng 15/09/05 14:50:29 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng) 15/09/05 14:50:29 INFO spark.HttpServer: Starting HTTP Server 15/09/05 14:50:30 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/09/05 14:50:30 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:44520 15/09/05 14:50:30 INFO util.Utils: Successfully started service 'HTTP class server' on port 44520. Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ version 1.4.0/_/Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79) Type in expressions to have them evaluated. Type :help for more information. 15/09/05 14:50:41 INFO spark.SparkContext: Running Spark version 1.4.0 15/09/05 14:50:41 INFO spark.SecurityManager: Changing view acls to: jifeng 15/09/05 14:50:41 INFO spark.SecurityManager: Changing modify acls to: jifeng 15/09/05 14:50:41 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng) 15/09/05 14:50:41 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/09/05 14:50:42 INFO Remoting: Starting remoting 15/09/05 14:50:42 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.110:45394] 15/09/05 14:50:42 INFO util.Utils: Successfully started service 'sparkDriver' on port 45394. 15/09/05 14:50:42 INFO spark.SparkEnv: Registering MapOutputTracker 15/09/05 14:50:43 INFO spark.SparkEnv: Registering BlockManagerMaster 15/09/05 14:50:43 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-21bdebec-d78f-4491-bd03-52defdf21f68/blockmgr-7206fb4f-673f-40dd-b8dd-3b178ee39938 15/09/05 14:50:43 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB 15/09/05 14:50:43 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-21bdebec-d78f-4491-bd03-52defdf21f68/httpd-8c3af065-b7fe-4fb4-a5bd-8dde0b236698 15/09/05 14:50:43 INFO spark.HttpServer: Starting HTTP Server 15/09/05 14:50:43 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/09/05 14:50:43 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:37858 15/09/05 14:50:43 INFO util.Utils: Successfully started service 'HTTP file server' on port 37858. 15/09/05 14:50:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/09/05 14:50:44 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/09/05 14:50:44 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/09/05 14:50:44 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/09/05 14:50:44 INFO ui.SparkUI: Started SparkUI at http://192.168.0.110:4040 15/09/05 14:50:45 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@feng03:7077/user/Master... 15/09/05 14:50:45 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150905145045-0000 15/09/05 14:50:46 INFO client.AppClient$ClientActor: Executor added: app-20150905145045-0000/0 on worker-20150905101207-192.168.0.110-58490 (192.168.0.110:58490) with 1 cores 15/09/05 14:50:46 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20150905145045-0000/0 on hostPort 192.168.0.110:58490 with 1 cores, 512.0 MB RAM 15/09/05 14:50:46 INFO client.AppClient$ClientActor: Executor updated: app-20150905145045-0000/0 is now RUNNING 15/09/05 14:50:46 INFO client.AppClient$ClientActor: Executor updated: app-20150905145045-0000/0 is now LOADING 15/09/05 14:50:46 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57911. 15/09/05 14:50:46 INFO netty.NettyBlockTransferService: Server created on 57911 15/09/05 14:50:46 INFO storage.BlockManagerMaster: Trying to register BlockManager 15/09/05 14:50:46 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.0.110:57911 with 267.3 MB RAM, BlockManagerId(driver, 192.168.0.110, 57911) 15/09/05 14:50:46 INFO storage.BlockManagerMaster: Registered BlockManager 15/09/05 14:50:47 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 15/09/05 14:50:47 INFO repl.SparkILoop: Created spark context.. Spark context available as sc. 15/09/05 14:50:50 INFO hive.HiveContext: Initializing execution hive, version 0.13.1 java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "1s"at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:105)at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:167)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)at $iwC$$iwC.<init>(<console>:9)at $iwC.<init>(<console>:18)at <init>(<console>:20)at .<init>(<console>:24)at .<clinit>(<console>)at .<init>(<console>:7)at .<clinit>(<console>)at $print(<console>)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:130)at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)at org.apache.spark.repl.Main$.main(Main.scala:31)at org.apache.spark.repl.Main.main(Main.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.NumberFormatException: For input string: "1s"at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)at java.lang.Integer.parseInt(Integer.java:492)at java.lang.Integer.parseInt(Integer.java:527)at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134)at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)... 56 more15/09/05 14:50:58 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.0.110:44198/user/Executor#934818115]) with ID 0 15/09/05 14:50:59 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.0.110:35246 with 267.3 MB RAM, BlockManagerId(0, 192.168.0.110, 35246) <console>:10: error: not found: value sqlContextimport sqlContext.implicits._^ <console>:10: error: not found: value sqlContextimport sqlContext.sql^scala> exit
總結(jié)
以上是生活随笔為你收集整理的Spark1.4 和 Hive 1.1.1 启动错误的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。
- 上一篇： Hive 1.1.1 启动错误
- 下一篇： Spark1.2新特性概述