1
저는 Scala와 Spark를 처음 사용합니다.왜 lines.map는 작동하지 않지만 lines.take.map은 Spark에서 작동합니까?
나는 SparkHdfsLR.scala code으로 연습하고 있습니다.
는하지만이 부분의 코드 문제로 실행 :60 val lines = sc.textFile(inputPath)
61 val points = lines.map(parsePoint _).cache()
62 val ITERATIONS = args(2).toInt
라인 (61)이 작동하지 않습니다. 나는이로 변경 후 :
60 val lines = sc.textFile(inputPath)
61 val points = lines.take(149800).map(parsePoint _) //149800 is the total number of lines
62 val ITERATIONS = args(2).toInt
벽성 SBT 실행에서 오류 MSG는 다음과 같습니다
[error] (run-main) org.apache.spark.SparkException: Job failed: Task 0.0:1 failed more than 4 times
org.apache.spark.SparkException: Job failed: Task 0.0:1 failed more than 4 times
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)
at org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[error] {file:/var/sdb/home/tim.tan/workspace/spark/}default-d3d73f/compile:run: Nonzero exit code: 1
[error] Total time: 52 s, completed Dec 20, 2013 5:42:18 PM
작업 노드의 표준 오차는 다음과 같습니다 작업자
13/12/20 17:42:16 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/20 17:42:16 INFO executor.StandaloneExecutorBackend: Connecting to driver: akka://[email protected]:38975/user/StandaloneScheduler
13/12/20 17:42:17 INFO executor.StandaloneExecutorBackend: Successfully registered with driver
13/12/20 17:42:17 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/20 17:42:17 INFO spark.SparkEnv: Connecting to BlockManagerMaster: akka://[email protected]:38975/user/BlockManagerMaster
13/12/20 17:42:17 INFO storage.MemoryStore: MemoryStore started with capacity 323.9 MB.
13/12/20 17:42:17 INFO storage.DiskStore: Created local directory at /tmp/spark-local-20131220174217-be8e
13/12/20 17:42:17 INFO network.ConnectionManager: Bound socket to port 52043 with id = ConnectionManagerId(TS-BH90,52043)
13/12/20 17:42:17 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/12/20 17:42:17 INFO storage.BlockManagerMaster: Registered BlockManager
13/12/20 17:42:17 INFO spark.SparkEnv: Connecting to MapOutputTracker: akka://[email protected]:38975/user/MapOutputTracker
13/12/20 17:42:17 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-1b1a6c0b-965e-4834-a3d3-554c95442041
13/12/20 17:42:17 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/20 17:42:17 INFO server.AbstractConnector: Started [email protected]:41811
13/12/20 17:42:18 ERROR executor.StandaloneExecutorBackend: Driver terminated or disconnected! Shutting down.
로그 다음과 같습니다 :
13/12/19 17:49:26 INFO worker.Worker: Asked to launch executor app-20131219174926-0001/2 for SparkHdfsLR
13/12/19 17:49:26 INFO worker.ExecutorRunner: Launch command: "java" "-cp" ":/var/bh/spark/conf:/var/bh/spark/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop1.0.3.jar:/var/bh/spark/core/target/scala-2.9.3/test-classes:/var/bh/spark/repl/target/scala-2.9.3/test-classes:/var/bh/spark/mllib/target/scala-2.9.3/test-classes:/var/bh/spark/bagel/target/scala-2.9.3/test-classes:/var/bh/spark/streaming/target/scala-2.9.3/test-classes" "-Djava.library.path=/var/bh/hadoop/lib/native/Linux-amd64-64/" "-Xms512M" "-Xmx512M" "org.apache.spark.executor.StandaloneExecutorBackend" "akka://[email protected]:56158/user/StandaloneScheduler" "2" "TS-BH87" "8"
13/12/19 17:49:30 INFO worker.Worker: Asked to kill executor app-20131219174926-0001/2
13/12/19 17:49:30 INFO worker.ExecutorRunner: Runner thread for executor app-20131219174926-0001/2 interrupted
13/12/19 17:49:30 INFO worker.ExecutorRunner: Killing process!
e 작업자로드가 성공적으로 시작되지 않았습니다.
이유를 모르겠습니다. 나에게 제안을 줄 수있는 사람이 있습니까?
줄의 종류를 지정하십시오. – senia
@senia [RDD] (https://spark.incubator.apache.org/docs/0.6.0/api/core/spark/RDD.html) –
"작동하지 않음"이란 무엇을 의미합니까? –