1. 홈
  2. 질의 응답
  3. 문제는

문제는

2016-08-22 11 views
0

는 스파크 2.0을 원했고,하지만 난 http://spark.apache.org/docs/latest/sql-programming-guide.html#getting-started문제는

import org.apache.spark.sql.SparkSession 
import spark.implicits._ 

val peopleDF = spark.read.format("json").load("myDir/examples/src/main/resources/people.json")

에서 다음 코드를 실행하려고 할 때 나는 그만 둘 :

16/08/22 22:59:53 WARN Hive: Failed to access metastore. This class should not accessed in runtime. 
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.Sess 
ionHiveMetaStoreClient 
     at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) 
     at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) 
     at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) 
     at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) 
     at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:171) 
     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
     at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) 
     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) 
     at java.lang.reflect.Constructor.newInstance(Unknown Source) 
     at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258) 
     at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359) 
     at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) 
     at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) 
     at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) 
     at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46) 
     at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45) 
     at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50) 
     at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48) 
     at org.apache.spark.sql.hive.HiveSessionState$$anon$1.<init>(HiveSessionState.scala:63) 
     at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63) 
     at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62) 
     at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49) 
     at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64) 
     at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:382) 
     at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:143) 
     at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:132) 
     at $line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27) 
     at $line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:32) 
     at $line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:34) 
     at $line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:36) 
     at $line19.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38) 
     at $line19.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:40) 
     at $line19.$read$$iw$$iw$$iw$$iw.<init>(<console>:42) 
     at $line19.$read$$iw$$iw$$iw.<init>(<console>:44) 
     at $line19.$read$$iw$$iw.<init>(<console>:46) 
     at $line19.$read$$iw.<init>(<console>:48) 
     at $line19.$read.<init>(<console>:50) 
     at $line19.$read$.<init>(<console>:54) 
     at $line19.$read$.<clinit>(<console>) 
     at $line19.$eval$.$print$lzycompute(<console>:7) 
     at $line19.$eval$.$print(<console>:6) 
     at $line19.$eval.$print(<console>) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
     at java.lang.reflect.Method.invoke(Unknown Source) 
     at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) 
     at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) 
     at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638) 
     at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637) 
     at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) 
     at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19) 
     at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637) 
     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) 
     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) 
     at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) 
     at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) 
     at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) 
     at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:415) 
     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:923) 
     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) 
     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) 
     at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) 
     at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) 
     at org.apache.spark.repl.Main$.doMain(Main.scala:68) 
     at org.apache.spark.repl.Main$.main(Main.scala:51) 
     at org.apache.spark.repl.Main.main(Main.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
     at java.lang.reflect.Method.invoke(Unknown Source) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 
     at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) 
     at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) 
     at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) 
     at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) 
     at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) 
     at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) 
     at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234) 
     ... 75 more 

Caused by: java.lang.reflect.InvocationTargetException 
     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
     at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) 
     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) 
     at java.lang.reflect.Constructor.newInstance(Unknown Source) 
     at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) 
     ... 81 more 

Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:/Users/Woahstan/spark-war 
ehouse 
     at org.apache.hadoop.fs.Path.initialize(Path.java:206) 
     at org.apache.hadoop.fs.Path.<init>(Path.java:172) 
     at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159) 
     at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:177) 
     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:600) 
     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620) 
     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461) 
     at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66) 
     at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) 
     at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762) 
     at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199) 
     at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) 
     ... 86 more 

Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:C:/Users/Woahstan/spark-warehouse 
     at java.net.URI.checkPath(Unknown Source) 
     at java.net.URI.<init>(Unknown Source) 
     at org.apache.hadoop.fs.Path.initialize(Path.java:203) 
     ... 97 more 

16/08/22 22:59:53 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 

    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) 
    at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:171) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) 
    at java.lang.reflect.Constructor.newInstance(Unknown Source) 
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258) 
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359) 
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) 
    at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) 
    at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) 
    at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46) 
    at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45) 
    at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50) 
    at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48) 
    at org.apache.spark.sql.hive.HiveSessionState$$anon$1.<init>(HiveSessionState.scala:63) 
    at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63) 
    at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62) 
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49) 
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64) 
    at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:382) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:143) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:132) 
    ... 50 elided 

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) 
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) 
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) 
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) 
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) 
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) 
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) 
    ... 72 more 

Caused by: java.lang.reflect.InvocationTargetException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in a 
bsolute URI: file:C:/Users/Woahstan/spark-warehouse 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) 
    at java.lang.reflect.Constructor.newInstance(Unknown Source) 
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) 
    ... 78 more 

Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:/Users/Woahstan/spark-war 
ehouse 
    at org.apache.hadoop.fs.Path.initialize(Path.java:206) 
    at org.apache.hadoop.fs.Path.<init>(Path.java:172) 
    at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159) 
    at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:177) 
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:600) 
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620) 
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461) 
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66) 
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) 
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762) 
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199) 
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) 
    ... 83 more 

Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:C:/Users/Woahstan/spark-warehouse 
    at java.net.URI.checkPath(Unknown Source) 
    at java.net.URI.<init>(Unknown Source) 
    at org.apache.hadoop.fs.Path.initialize(Path.java:203) 
    ... 94 more

어떤 아이디어 무엇이 잘못되었으며 어떻게 해결할 수 있습니까?

출처

2016-08-22 Ayu

+0

정확한 오류 메시지를 확인할 수 있습니까? Caused by : 오류 메시지에서 scala 프롬프트가 나타나기 전에 몇 줄을 확인하십시오. – abaghel

+0

편집 * : 이것은 표시됩니다. – Ayu

답변

0

Spark 2.0.0의 알려진 문제 (https://issues.apache.org/jira/browse/SPARK-15899)입니다. 그런 다음 사용 스파크 세션과 2.0.0에서 자바 예제를 실행하려면 below-

spark-shell --conf spark.sql.warehouse.dir=file:///C:/tmp/spark-warehouse

처럼 --conf 인수와 함께 불꽃 쉘을 시작합니다.

SparkSession spark = SparkSession 
    .builder() 
    .appName("JavaALSExample") 
    .config("spark.sql.warehouse.dir", "file:///C:/tmp/spark-warehouse") 
    .master("local[2]") 
    .getOrCreate();

출처

2016-08-23 06:46:35 abaghel

+0

감사합니다. --conf는이 문제를 해결하는데 정말로 도움이되었습니다. – Ayu

관련 문제

 관련 문제