1. 홈
  2. 질의 응답
  3. 는 ...이처럼 보이는 정말 간단한 작업이 윈도우

는 ...이처럼 보이는 정말 간단한 작업이 윈도우

2016-12-22 1 views
1

에 심지어 간단한 스파크 수를 실행할 수 없습니다는 ...이처럼 보이는 정말 간단한 작업이 윈도우

object FileCount { 
    def main(args: Array[String]) { 
    val sc = new SparkContext(new SparkConf().setAppName("Spark File Count").setMaster("local[8]")) 
    val rdd = sc.parallelize(List("a", "b", "c", "d", "e")) 
    val count = rdd.count() 
    println(s"Count $count") 
    sc.stop() 
    } 
}

즉이 정말 추하고 인정 스택 추적 결과 ...

16/12/21 17:05:45 INFO Executor: Fetching http://myipaddress:59850/jars/file-count-0.0.1-SNAPSHOT.jar with timestamp 1482357926972 
16/12/21 17:05:45 INFO Utils: Fetching http://myipaddress:59850/jars/file-count-0.0.1-SNAPSHOT.jar to C:\Users\579912\AppData\Local\Temp\spark-2d04e486-3904-4c4b-ba1f-20c5d49049ff\userFiles-e72fd11f-6bf7-40d6-846a-ee86a59f4f09\fetchFileTemp6527719131307552432.tmp 
16/12/21 17:05:45 INFO Executor: Fetching http://myipaddress:59850/jars/file-count-0.0.1-SNAPSHOT.jar with timestamp 1482357926972 
16/12/21 17:05:45 ERROR Executor: Exception in task 2.0 in stage 0.0 (TID 2) 
java.lang.NullPointerException 
     at java.lang.ProcessBuilder.start(Unknown Source) 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:482) 
     at org.apache.hadoop.util.Shell.run(Shell.java:455) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) 
     at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:430) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:422) 
     at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) 
     at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
     at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
     at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) 
     at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:422) 
     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
     at java.lang.Thread.run(Unknown Source) 
16/12/21 17:05:45 INFO Utils: Fetching http://myipaddress:59850/jars/file-count-0.0.1-SNAPSHOT.jar to C:\Users\579912\AppData\Local\Temp\spark-2d04e486-3904-4c4b-ba1f-20c5d49049ff\userFiles-e72fd11f-6bf7-40d6-846a-ee86a59f4f09\fetchFileTemp7055275334836023373.tmp 
16/12/21 17:05:45 INFO Utils: C:\Users\579912\AppData\Local\Temp\spark-2d04e486-3904-4c4b-ba1f-20c5d49049ff\userFiles-e72fd11f-6bf7-40d6-846a-ee86a59f4f09\fetchFileTemp7055275334836023373.tmp has been previously copied to C:\Users\579912\AppData\Local\Temp\spark-2d04e486-3904-4c4b-ba1f-20c5d49049ff\userFiles-e72fd11f-6bf7-40d6-846a-ee86a59f4f09\file-count-0.0.1-SNAPSHOT.jar 
16/12/21 17:05:45 WARN TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2, localhost): java.lang.NullPointerException 
     at java.lang.ProcessBuilder.start(Unknown Source) 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:482) 
     at org.apache.hadoop.util.Shell.run(Shell.java:455) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) 
     at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:430) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:422) 
     at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) 
     at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
     at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
     at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) 
     at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:422) 
     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
     at java.lang.Thread.run(Unknown Source) 

16/12/21 17:05:45 INFO Executor: Fetching http://myipaddress:59850/jars/file-count-0.0.1-SNAPSHOT.jar with timestamp 1482357926972 
16/12/21 17:05:45 ERROR Executor: Exception in task 4.0 in stage 0.0 (TID 4) 
java.lang.NullPointerException 
     at java.lang.ProcessBuilder.start(Unknown Source) 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:482) 
     at org.apache.hadoop.util.Shell.run(Shell.java:455) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) 
     at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:430) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:422) 
     at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) 
     at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
     at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
     at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) 
     at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:422) 
     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
     at java.lang.Thread.run(Unknown Source) 
16/12/21 17:05:45 ERROR TaskSetManager: Task 2 in stage 0.0 failed 1 times; aborting job 
16/12/21 17:05:45 INFO TaskSetManager: Lost task 4.0 in stage 0.0 (TID 4) on executor localhost: java.lang.NullPointerException (null) [duplicate 1] 
16/12/21 17:05:45 INFO TaskSchedulerImpl: Cancelling stage 0 
16/12/21 17:05:45 INFO Executor: Executor is trying to kill task 0.0 in stage 0.0 (TID 0) 
16/12/21 17:05:45 INFO TaskSchedulerImpl: Stage 0 was cancelled 
16/12/21 17:05:45 INFO Executor: Executor is trying to kill task 1.0 in stage 0.0 (TID 1) 
16/12/21 17:05:45 INFO Utils: Fetching http://myipaddress:59850/jars/file-count-0.0.1-SNAPSHOT.jar to C:\Users\579912\AppData\Local\Temp\spark-2d04e486-3904-4c4b-ba1f-20c5d49049ff\userFiles-e72fd11f-6bf7-40d6-846a-ee86a59f4f09\fetchFileTemp8100007898534254285.tmp 
16/12/21 17:05:45 INFO DAGScheduler: ResultStage 0 (count at FileCount.scala:19) failed in 0.451 s 
16/12/21 17:05:45 INFO Executor: Executor is trying to kill task 5.0 in stage 0.0 (TID 5) 
16/12/21 17:05:45 INFO Executor: Executor is trying to kill task 6.0 in stage 0.0 (TID 6) 
16/12/21 17:05:45 INFO Executor: Executor is trying to kill task 3.0 in stage 0.0 (TID 3) 
16/12/21 17:05:45 INFO Executor: Executor is trying to kill task 7.0 in stage 0.0 (TID 7) 
16/12/21 17:05:45 INFO DAGScheduler: Job 0 failed: count at FileCount.scala:19, took 0.847770 s 
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2, localhost): java.lang.NullPointerException 
     at java.lang.ProcessBuilder.start(Unknown Source) 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:482) 
     at org.apache.hadoop.util.Shell.run(Shell.java:455) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) 
     at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:430) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:422) 
     at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) 
     at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
     at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
     at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) 
     at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:422) 
     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
     at java.lang.Thread.run(Unknown Source) 

Driver stacktrace: 
     at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) 
     at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) 
     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) 
     at scala.Option.foreach(Option.scala:236) 
     at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) 
     at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 
     at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858) 
     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929) 
     at org.apache.spark.rdd.RDD.count(RDD.scala:1157) 
     at FileCount$.main(FileCount.scala:19) 
     at FileCount.main(FileCount.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
     at java.lang.reflect.Method.invoke(Unknown Source) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: java.lang.NullPointerException 
     at java.lang.ProcessBuilder.start(Unknown Source) 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:482) 
     at org.apache.hadoop.util.Shell.run(Shell.java:455) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873) 
     at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) 
     at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:430) 
     at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:422) 
     at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
     at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) 
     at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
     at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
     at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) 
     at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:422) 
     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
     at java.lang.Thread.run(Unknown Source)

아이디어가 있으십니까? 나는 이것을 다음과 같이 실행한다. spark-submit --master local[8] --class FileCount --verbose target/file-count.jar

출처

2016-12-22 Mike Thomsen

+0

Windows에서 알려진 문제입니다. http://stackoverflow.com/questions/27201505/hadoop-exception-in-thread-main-java-lang-nullpointerexception –

+0

더 정확히 말하자면 여기 문서가 수정 된 것 같습니다. https://wiki.apache.org/hadoop/WindowsProblems 및 여기 # 5 단계 - http://stackoverflow.com/a/38735202/2308683 –

답변

0

this을 추천하는 코멘트는 나를 위해 완벽하게 작동했다. 문제가 완전히 해결되었습니다.

출처

2016-12-22 00:58:11

관련 문제

 관련 문제