3
안녕하세요, 저는 단순한 2 노드 클러스터 1 마스터와 1 슬레이브 (hadoop 및 spark)를 보유하고 있습니다. 모든 것이 잘 작동했지만 마스터 스파크를 실행할 때 어떤 이유로 마스터 웹 UI (포트 8081을 spark-env.sh 파일에 명시 적으로 설정)에 연결할 수 없습니다.아파치 스파크 마스터 ui 작동하지 않음
Spark Command: /usr/lib/jvm/java-8-oracle/bin/java -cp /usr/local/hive/lib/*:/usr/local/spark/conf/:/usr/local/spark/assembly/target/scala-2.11/jars/*:/usr/local/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host ip-172-31-18-164 --port 7077 --webui-port 8081
========================================
16/07/23 00:47:55 INFO master.Master: Started daemon with process name: [email protected]
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for TERM
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for HUP
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for INT
16/07/23 00:47:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls groups to:
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls groups to:
16/07/23 00:47:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); groups with view permissions: Set(); users with modify permissions: Set(ubuntu); groups with modify permissions: Set()
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
16/07/23 00:47:56 INFO master.Master: Starting Spark master at spark://ip-172-31-18-164:7077
16/07/23 00:47:56 INFO master.Master: Running Spark version 2.1.0-SNAPSHOT
16/07/23 00:47:56 INFO util.log: Logging initialized @1447ms
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/app,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/app/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/app/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/driver/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8081}
16/07/23 00:47:56 INFO server.Server: Started @1558ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'MasterUI' on port 8081.
16/07/23 00:47:56 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://172.31.18.164:8081
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{ip-172-31-18-164:6066}
16/07/23 00:47:56 INFO server.Server: Started @1587ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service on port 6066.
16/07/23 00:47:56 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/metrics/master/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started [email protected]{/metrics/applications/json,null,AVAILABLE}
16/07/23 00:47:56 INFO master.Master: I have been elected leader! New state: ALIVE
16/07/23 00:47:57 WARN server.HttpChannel:/
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)
at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.spark_project.jetty.server.Server.handle(Server.java:499)
at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool:
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getStatus()I
at org.spark_project.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:112)
at org.spark_project.jetty.server.Response.sendError(Response.java:597)
at org.spark_project.jetty.server.HttpChannel.handleException(HttpChannel.java:487)
at org.spark_project.jetty.server.HttpConnection$HttpChannelOverHttp.handleException(HttpConnection.java:594)
at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:387)
at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool: Unexpected thread death: [email protected] in MasterUI{STARTED,8<=8<=200,i=5,q=0}
16/07/23 00:47:58 INFO master.Master: Registering worker 172.31.7.238:60863 with 2 cores, 6.0 GB RAM
그래 나는 내가 mysql.jar를 얻기 위해 하이브 리브 내 SPARK_CLASSPATH에서 *를하고 있던 것을 깨달았 그러나 그것은 서블릿의 다른 버전에서 초래 된 . 나는 명시 적으로 mysql.jar 파일을 명시 적으로 호출했다. – Neil