스파크 작업이 Google Dataproc과 호환되지 않음

Question

한 줄을 변경하지 않았지만이 코드는 몇 개월 동안 정상적으로 작동했으며 약 2 개월 전 Google Dataproc을 사용하여 작업을 중단했습니다.

WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, test-cluster-w-1.c.test-project.internal): java.lang.AbstractMethodError: uk.co.test.CalculateScore$$Lambda$10/1666820030.call(Ljava/lang/Object;)Ljava/util/Iterator; 
     at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:142) 
     at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:142) 
     at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) 
     at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) 
     at scala.collection.Iterator$class.foreach(Iterator.scala:893) 
     at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) 
     at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) 
     at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) 
     at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) 
     at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) 
     at scala.collection.AbstractIterator.to(Iterator.scala:1336) 
     at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) 
     at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336) 
     at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) 
     at scala.collection.AbstractIterator.toArray(Iterator.scala:1336) 
     at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:893) 
     at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:893) 
     at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1897) 
     at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1897) 
     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 
     at org.apache.spark.scheduler.Task.run(Task.scala:85) 
     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
     at java.lang.Thread.run(Thread.java:745) 

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, test-cluster-w-0.c.test-project.internal): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container marked as failed: container_1475077182957_0001_01_000005 on host: sun-recommendations-evaluation-w-0.c.test-project.internal. Exit status: 50. Diagnostics: Exception from container-launch. 
Container id: container_1475077182957_0001_01_000005 
Exit code: 50 
Stack trace: ExitCodeException exitCode=50: 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) 
     at org.apache.hadoop.util.Shell.run(Shell.java:456) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) 
     at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) 
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) 
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) 
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
     at java.lang.Thread.run(Thread.java:745) 


Container exited with a non-zero exit code 50 

Driver stacktrace: 
     at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437) 
     at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) 
     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811) 
     at scala.Option.foreach(Option.scala:257) 
     at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607) 
     at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 
     at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1897) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1911) 
     at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:893) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) 
     at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) 
     at org.apache.spark.rdd.RDD.collect(RDD.scala:892) 
     at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:360) 
     at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45) 
     at uk.co.test.CalculateScore.main(CalculateScore.java:50) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:498) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

경우 :

SparkConf sparkConf = new SparkConf().setAppName("test"); 
JavaSparkContext jsc = new JavaSparkContext(sparkConf); 

JavaRDD<String> rdd = jsc.parallelize(Arrays.asList("a", "b", "c")); 
JavaPairRDD<String, String> pairs = rdd.flatMapToPair(value -> 
     Arrays.asList(
       new Tuple2<>(value, value + "1"), 
       new Tuple2<>(value, value + "2") 
     ) 
); 
pairs.collect().forEach(System.out::println); 

가 그럼이 모호한 예외를 얻을 :

난 그냥 몇 줄의 버그를 재현 할 수

그래서 코드의 거대한 블록을 게시하지 않았다 그 다음은 잘 작동하고 출력

sparkConf.setMaster("local[2]") 

: 나는 로컬로 실행

,369 어떤 도움에 감사드립니다

<dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-mllib_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 

:

(a,a1) 
(a,a2) 
(b,b1) 
(b,b2) 
(c,c1) 
(c,c2) 

이 1,363,210 나의 불꽃으로 종속되어 있습니다.

Answer 1

Dataproc에서 사용하는 기본 이미지는 최근에 Spark 2.0/Scala 2.11로 업그레이드되었습니다. 이것은 8 월에 변경되어 잠재적으로 차이를 설명 할 수 있습니다.

This page 각 Dataproc 이미지 릴리스에 어떤 버전의 소프트웨어 패키지가 포함되어 있는지 자세히 설명합니다.

그것은, 다음으로 pom.xml 파일을 업데이트 다시 컴파일하기에 충분할 수있다

<dependency> 
    <groupId>org.apache.spark</groupId> 
    <artifactId>spark-core_2.11</artifactId> 
    <version>2.0.0</version> 
</dependency> 
<dependency> 
    <groupId>org.apache.spark</groupId> 
    <artifactId>spark-mllib_2.11</artifactId> 
    <version>2.0.0</version> 
</dependency> 
<dependency> 
    <groupId>org.apache.spark</groupId> 
    <artifactId>spark-streaming_2.11</artifactId> 
    <version>2.0.0</version> 
</dependency> 

스파크 2.0의 release notes는 스파크 1.6과 2.0 사이의 changes and removals가 포함 된 응용 프로그램을 다시 실행합니다. 대안으로

, 당신은 여전히 ​​다음 gcloud 호출을 사용하여 1.0 이미지 트랙을 사용할 수있다 :, 명시 적 이미지 트랙을 사용하는 경우

$ gcloud dataproc clusters create --image-version 1.0 ... 

를 주/부 버전은 더 이상 사용되지 될 수 있음을 명심하고 결국 제거되었습니다. 이미지 버전에 대한 지원 일정은 Dataproc image versioning 정책을 참조하십시오.