저는 Hadoop에서 절대 초보자입니다. 그러나 간단한 테스트 만하고 있습니다. 그러나 오류 메시지는 매우 유익하지 않습니다.Hadoop : 돼지 오류
4Gb RAM을 사용할 수있는 CentOS 6.4 VM에서 단일 노드 모드로 Hadoop 환경을 설정했습니다.
500Mb CSV 파일을 통해 간단한 돼지 스크립트를 실행하려고합니다. 두 개의 500MB 파일을 가지고 있는데 첫 번째 파일은 스크립트가 성공한 것입니다. 둘째, 같은 크기이지만 다른 데이터 (더 많은 행)에 대해서는 실행이 약 60 %에 도달하면 오류가 발생합니다. 내가 터미널에서 얻을 오류
records = LOAD 'trans2013.csv' USING PigStorage(',') AS
(podracun_v_breme,datum_transakcije,znesek_transakcije,oznaka_valute_transakcije,racun_v_dobro,naziv_prejemnika,maticna_stevilka,davcna_stevilka,sifra_pu,zr_sns_oe,namen);
transaction_recs = GROUP records ALL;
tot_trans = FOREACH transaction_recs GENERATE
SUM(records.znesek_transakcije);
STORE tot_trans INTO '/user/root/totaltransactions';
이것은 : 여기
2014-04-06 10:28:29,147 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 64% complete 2014-04-06 10:28:30,240 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. 2014-04-06 10:28:30,241 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1396637732046_0008 has failed! Stop running all dependent jobs 2014-04-06 10:28:30,241 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2014-04-06 10:28:30,460 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
2014-04-06 10:28:30,461 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! 2014-04-06 10:28:30,463 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.0.6-alpha 0.11.1 root 2014-04-06 10:25:49 2014-04-06 10:28:30 GROUP_BY
Failed!
Failed Jobs: JobId Alias Feature Message Outputs job_1396637732046_0008 records,tot_trans,transaction_recs GROUP_BY,COMBINER Message: Job failed! /user/root/totaltransactions,
Input(s): Failed to read data from "hdfs://localhost:8020/user/root/trans2013.csv"
Output(s): Failed to produce result in "/user/root/totaltransactions"
Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0
Job DAG: job_1396637732046_0008
2014-04-06 10:28:30,463 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! 2014-04-06 10:28:30,491 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
Details at logfile: /root/pig_1396797945352.log
로그에서 오류는 다음과 같습니다
이
은 내가 사용하는 (매우 간단) 돼지 스크립트입니다Backend error message --------------------- AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
Pig Stack Trace --------------- ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:217) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:149) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:400) at org.apache.pig.PigServer.launchPlan(PigServer.java:1264) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249) at org.apache.pig.PigServer.execute(PigServer.java:1239) at org.apache.pig.PigServer.executeBatch(PigServer.java:333) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:604) at org.apache.pig.Main.main(Main.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)