1
Intellij에서 spark master와 worker를 실행하고 싶습니다. 나는 spark master와 worker를 성공적으로 시작했다. 작업자는 아무런 문제없이 마스터와도 연결됩니다. 로그를보고 웹 UI를 실행하여이를 확인할 수 있습니다. 그러나이 설정에서 응용 프로그램을 실행하려고하면 문제가 시작됩니다. 현재,이 설치 프로그램에서 spark-shell을 실행하려고 시도하고 있지만 마스터하려고하면 작업자에서 IlligalStateException 및 Master에 대한 오류 메시지가 나타납니다.java.lang.IllegalStateException : 빌드 디렉토리를 찾을 수 없습니다.
마스터 로그
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/09/21 16:42:33 INFO Master: Started daemon with process name: [email protected]
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for TERM
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for HUP
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for INT
16/09/21 16:42:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/21 16:42:34 INFO SecurityManager: Changing view acls to: rana
16/09/21 16:42:34 INFO SecurityManager: Changing modify acls to: rana
16/09/21 16:42:34 INFO SecurityManager: Changing view acls groups to:
16/09/21 16:42:34 INFO SecurityManager: Changing modify acls groups to:
16/09/21 16:42:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 16:42:35 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
16/09/21 16:42:35 INFO Master: Starting Spark master at spark://MASTER_IP:7077
16/09/21 16:42:35 INFO Master: Running Spark version 2.0.0
16/09/21 16:42:35 INFO log: Logging initialized @3645ms
16/09/21 16:42:36 INFO Server: jetty-9.2.16.v20160414
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/app,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/app/json,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/json,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/static,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/app/kill,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/driver/kill,null,AVAILABLE}
16/09/21 16:42:36 INFO ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8080}
16/09/21 16:42:36 INFO Server: Started @3999ms
16/09/21 16:42:36 INFO Utils: Successfully started service 'MasterUI' on port 8080.
16/09/21 16:42:36 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://MASTER_IP:8080
16/09/21 16:42:36 INFO Server: jetty-9.2.16.v20160414
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/09/21 16:42:36 INFO ServerConnector: Started [email protected]{HTTP/1.1}{MASTER_IP:6066}
16/09/21 16:42:36 INFO Server: Started @4050ms
16/09/21 16:42:36 INFO Utils: Successfully started service on port 6066.
16/09/21 16:42:36 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/metrics/master/json,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started [email protected]{/metrics/applications/json,null,AVAILABLE}
16/09/21 16:42:36 INFO Master: I have been elected leader! New state: ALIVE
16/09/21 16:42:46 INFO Master: Registering worker MASTER_IP:42839 with 6 cores, 6.7 GB RAM
노동자 로그
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/09/21 17:24:08 INFO Worker: Started daemon with process name: [email protected]
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for TERM
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for HUP
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for INT
16/09/21 17:24:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/21 17:24:08 INFO SecurityManager: Changing view acls to: rana
16/09/21 17:24:08 INFO SecurityManager: Changing modify acls to: rana
16/09/21 17:24:08 INFO SecurityManager: Changing view acls groups to:
16/09/21 17:24:08 INFO SecurityManager: Changing modify acls groups to:
16/09/21 17:24:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 17:24:09 INFO Utils: Successfully started service 'sparkWorker' on port 50059.
16/09/21 17:24:09 INFO Worker: Starting Spark worker MASTER_IP:50059 with 6 cores, 6.7 GB RAM
16/09/21 17:24:09 INFO Worker: Running Spark version 2.0.0
16/09/21 17:24:09 INFO Worker: Spark home: /usr/local/spark
16/09/21 17:24:09 INFO log: Logging initialized @2328ms
16/09/21 17:24:09 INFO Server: jetty-9.2.16.v20160414
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/logPage,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/logPage/json,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/json,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/static,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/log,null,AVAILABLE}
16/09/21 17:24:09 INFO ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8081}
16/09/21 17:24:09 INFO Server: Started @2440ms
16/09/21 17:24:09 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
16/09/21 17:24:09 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://MASTER_IP:8081
16/09/21 17:24:09 INFO Worker: Connecting to master MASTER_IP:7077...
16/09/21 17:24:09 INFO ContextHandler: Started [email protected]{/metrics/json,null,AVAILABLE}
16/09/21 17:24:09 INFO TransportClientFactory: Successfully created connection to /MASTER_IP:7077 after 35 ms (0 ms spent in bootstraps)
16/09/21 17:24:10 INFO Worker: Successfully registered with master spark://MASTER_IP:7077
마스터 로그 (성공적으로 시작) (성공적으로 시작)
(시도는 스파크 쉘을 실행하는 경우)16/09/21 17:26:08 INFO Master: Registering app Spark shell
16/09/21 17:26:08 INFO Master: Registered app Spark shell with ID app-20160921172608-0000
16/09/21 17:26:08 INFO Master: Launching executor app-20160921172608-0000/0 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/0 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/1 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/1 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/2 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/2 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/3 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/3 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/4 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/4 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/5 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/5 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/6 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/6 because it is FAILED
노동자 로그 (시도는 스파크 쉘을 실행하는 경우)
16/09/21 17:26:09 INFO Worker: Asked to launch executor app-20160921172608-0000/0 for Spark shell
16/09/21 17:26:09 INFO SecurityManager: Changing view acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing view acls groups to:
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls groups to:
16/09/21 17:26:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 17:26:09 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47)
at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
16/09/21 17:26:09 INFO Worker: Executor app-20160921172608-0000/0 finished with state FAILED message java.lang.IllegalStateException: Cannot find any build directories.
16/09/21 17:26:09 INFO Worker: Asked to launch executor app-20160921172608-0000/1 for Spark shell
16/09/21 17:26:09 INFO SecurityManager: Changing view acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing view acls groups to:
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls groups to:
16/09/21 17:26:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 17:26:09 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47)
at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
누군가가 나에게이 문제를 해결하는 데 도움이 될 수 있습니다 ??