Spark Scala 2.11.8 Spark HbaseConnector 오류

Question

jark을 사용하여 kafka 스트림의 데이터를 저장하기 위해 Spark scala 2.11.8 hbase 커넥터를 사용하려고합니다. 그러나 저장하려고하면 다음 오류가 발생합니다. hortoworks에서 shc 커넥터를 사용하고 있습니다. 내 SBT 설정은 다음과 같습니다.

이 커넥터는 계속 지원됩니까? java.lang.ClassNotFoundException가 :

Exception in thread "streaming-job-executor-1" java.lang.NoClassDefFoundError: org/apache/spark/Logging 
    at java.lang.ClassLoader.defineClass1(Native Method) 
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763) 
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) 
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
    at java.lang.Class.getDeclaredConstructors0(Native Method) 
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) 
    at java.lang.Class.getConstructor0(Class.java:3075) 
    at java.lang.Class.newInstance(Class.java:412) 
    at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:427) 
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211) 
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:194) 
    at $line20.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$CTUMhbaseingest$.saveHbase$1(<console>:193) 
    at $line20.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$CTUMhbaseingest$.runBusinessLogicAndProduceOutput(<console>:295) 
    at $line20.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$CTUMhbaseingest$$anonfun$run$1.apply(<console>:312) 
    at $line20.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$CTUMhbaseingest$$anonfun$run$1.apply(<console>:311) 
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627) 
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627) 
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51) 
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) 
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) 
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415) 
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50) 
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) 
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) 
    at scala.util.Try$.apply(Try.scala:192) 
    at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39) 
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:245) 
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:245) 
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:245) 
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) 
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:244) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

의해 발생 : java.net.URLClassLoader.findClass (URLClassLoader.java에서 을 org.apache.spark.Logging 다음

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-core_2.11" % "2.0.1" % "provided", 
    "org.apache.spark" % "spark-sql_2.11" % "2.0.1" % "provided", 
    "org.apache.spark" % "spark-streaming_2.11" % "2.0.1" % "provided", 
    ("org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.0.1").exclude("org.spark-project.spark", "unused"), 
    "org.json4s" % "json4s-native_2.11" % "3.2.10", 
    "joda-time" % "joda-time" % "2.9.9", 
    "com.hortonworks" % "shc" % "1.1.1-2.1-s_2.11" 
) 

오류 : 381)

Answer 1

실행 프로그램이나 드라이버 클래스 경로에 문제가 있습니다. org.apache.spark.Logging은 Spark 버전 1.5.2 이하 버전에서 사용할 수 있습니다. 그러나 귀하의 libraryDependencies에서 spark 2.0.1을 사용하고 있습니다. 스파크 애플리케이션 UI에서 환경 메뉴를 체크하고 드라이버 & 실행자 클래스 경로를 볼 수 있습니다.