내 컴파일에서 스파이크 SBT 컴파일 문제

Question

src/main/resources 폴더에 twitter jar 파일을 배치하고 있지만 SBT 컴파일이 선택하지 않고 컴파일하고 오류없이 패키지하지만 런타임에 " 클래스를 찾을 수 없습니다 twitterUtils "

내 질문에 SBT는 컴파일에 리소스 폴더에서 jar 파일을 포함하지 않는 이유는 무엇입니까?

사람들은 Git 유틸리티를 얻은 다음 sbt 어셈블리를 수행하는 모든 복잡한 단계를 수행해야한다고 말하고 있습니다.하지만 모든 http_proxy 설치가 끝난 후에도 Proxy Git의 iam이 작동하지 않기 때문입니다. 나는 또한이 twitter jar 파일을 CLASSPATH에 넣어 보았습니다. 이 문제와 관련하여 도움을 드리고자합니다. SBT는 컴파일에 리소스 폴더에서 jar 파일을 포함하지 않는 이유는 내 질문은

[root@hadoop1 TwitterPopularTags]# pwd 
/root/TwitterPopularTags 
[root@hadoop1 TwitterPopularTags]# sbt compile 
[info] Set current project to TwitterPopularTags (in build file:/root/TwitterPopularTags/) 
[info] Updating {file:/root/TwitterPopularTags/}twitterpopulartags... 
[info] Resolving jline#jline;2.12.1 ... 
[info] Done updating. 
[info] Compiling 2 Scala sources to /root/TwitterPopularTags/target/scala-2.11/classes... 
[success] Total time: 14 s, completed Sep 16, 2016 9:55:20 AM 
[root@hadoop1 TwitterPopularTags]# sbt package 
[info] Set current project to TwitterPopularTags (in build file:/root/TwitterPopularTags/) 
[info] Packaging /root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar ... 
[info] Done packaging. 
[success] Total time: 1 s, completed Sep 16, 2016 9:56:20 AM 
[root@hadoop1 TwitterPopularTags]# spark-submit /root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar 
16/09/16 09:57:06 INFO SparkContext: Running Spark version 1.6.2 
16/09/16 09:57:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/09/16 09:57:06 INFO SecurityManager: Changing view acls to: root 
16/09/16 09:57:06 INFO SecurityManager: Changing modify acls to: root 
16/09/16 09:57:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 
16/09/16 09:57:07 INFO Utils: Successfully started service 'sparkDriver' on port 53967. 
16/09/16 09:57:07 INFO Slf4jLogger: Slf4jLogger started 
16/09/16 09:57:07 INFO Remoting: Starting remoting 
16/09/16 09:57:07 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.100.44.17:57877] 
16/09/16 09:57:07 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 57877. 
16/09/16 09:57:07 INFO SparkEnv: Registering MapOutputTracker 
16/09/16 09:57:07 INFO SparkEnv: Registering BlockManagerMaster 
16/09/16 09:57:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-47a89077-0926-447c-ada7-fdb4a9aa1b83 
16/09/16 09:57:07 INFO MemoryStore: MemoryStore started with capacity 511.5 MB 
16/09/16 09:57:07 INFO SparkEnv: Registering OutputCommitCoordinator 
16/09/16 09:57:08 INFO Server: jetty-8.y.z-SNAPSHOT 
16/09/16 09:57:08 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 
16/09/16 09:57:08 INFO Utils: Successfully started service 'SparkUI' on port 4040. 
16/09/16 09:57:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.100.44.17:4040 
16/09/16 09:57:08 INFO HttpFileServer: HTTP File server directory is /tmp/spark-d56628b6-fdbf-4d89-bbd2-a96603000607/httpd-ee499eb3-00ae-4276-b163-423e3b81f0b4 
16/09/16 09:57:08 INFO HttpServer: Starting HTTP Server 
16/09/16 09:57:08 INFO Server: jetty-8.y.z-SNAPSHOT 
16/09/16 09:57:08 INFO AbstractConnector: Started SocketConnector@0.0.0.0:56067 
16/09/16 09:57:08 INFO Utils: Successfully started service 'HTTP file server' on port 56067. 
16/09/16 09:57:08 INFO SparkContext: Added JAR file:/root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar at http://10.100.44.17:56067/jars/twitterpopulartags_2.11-1.0.jar with timestamp 1474034228091 
16/09/16 09:57:08 INFO Executor: Starting executor ID driver on host localhost 
16/09/16 09:57:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49715. 
16/09/16 09:57:08 INFO NettyBlockTransferService: Server created on 49715 
16/09/16 09:57:08 INFO BlockManagerMaster: Trying to register BlockManager 
16/09/16 09:57:08 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49715 with 511.5 MB RAM, BlockManagerId(driver, localhost, 49715) 
16/09/16 09:57:08 INFO BlockManagerMaster: Registered BlockManager 
16/09/16 09:57:08 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 
16/09/16 09:57:08 INFO EventLoggingListener: Logging events to hdfs:///spark-history/local-1474034228122 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ 
     at dot.state.fl.us.PrintTweets$.main(PrintTweets.scala:29) 
     at dot.state.fl.us.PrintTweets.main(PrintTweets.scala) 

Answer 1

아래의 내용을 참조하십시오?

리소스 폴더가 아니기 때문에. 의존성을 수동으로 관리하려면 put them into lib folder instead. 그러나이 경우 의존성, 종속성 등의 모든 종속성에 대해서도 동일한 작업을 수행해야합니다. 링크 된 문서에 설명 된대로 관리 종속성을 사용하면 훨씬 더 좋은 아이디어입니다.