MapReduce의 Java 힙 공간

Question

32GB RAM이있는 시스템에서 MapReduce 작업을 실행하고 있지만 Java 힙 공간 오류가 발생합니다. 나는 작업을 수행 할 수있는 충분한 메모리를 가지기를 희망하면서 32GB로 yarn.nodemanager.resource.memory-mb을 설정했습니다. 이 문제가 발생하지 않도록 MapReduce v2를 어떻게 설정해야합니까?

편집 :

이

16/08/30 19:00:49 INFO mapreduce.Job: Task Id : attempt_1472579604725_0003_m_000000_0, Status : FAILED 
Error: Java heap space 
16/08/30 19:00:55 INFO mapreduce.Job: Task Id : attempt_1472579604725_0003_m_000000_1, Status : FAILED 
Error: Java heap space 
16/08/30 19:01:00 INFO mapreduce.Job: Task Id : attempt_1472579604725_0003_m_000000_2, Status : FAILED 
Error: Java heap space 

[2] mapred-site.xml

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 

<!-- Put site-specific property overrides in this file. --> 
<configuration> 
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> 
<property> <name>mapreduce.jobhistory.done-dir</name> <value>/root/Programs/hadoop/logs/history/done</value> </property> 
<property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/root/Programs/hadoop/logs/history/intermediate-done-dir</value> </property> 
<property> <name>mapreduce.job.reduces</name> <value>2</value> </property> 

<!-- property> <name>yarn.nodemanager.resource.memory-mb</name> <value>10240</value> </property> 
<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property --> 

<!-- property><name>mapreduce.task.files.preserve.failedtasks</name><value>true</value></property> 
<property><name>mapreduce.task.files.preserve.filepattern</name><value>*</value></property --> 

[3] yarn-site.xml

<configuration> 
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> 
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> 
<property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> 
<property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>s8:8025</value> </property> 
<property> <name>yarn.resourcemanager.scheduler.address</name> <value>s8:8030</value> </property> 
<property> <name>yarn.resourcemanager.address</name> <value>s8:8032</value> </property> 
<property> <name>yarn.log.server.url</name> <value>http://s8:19888/jobhistory/logs/</value> </property> 

<!-- job history --> 
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> 
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property> 
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property> 

<!-- proxy --> 
<property><name>yarn.web-proxy.address</name><value>s8:9046</value></property> 

<!-- to check the classpath in yarn, do yarn classpath --> 
<!-- compress output data --> 
<property><name>mapreduce.output.fileoutputformat.compress</name><value>false</value></property> 
<property><name>mapred.output.fileoutputformat.compress.codec</name><value>org.apache.hadoop.io.compress.BZip2Codec</value></property> 

<!-- Node configuration --> 
    <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>33554432</value> </property> 
</configuration> 

Answer 1

매개 변수 yarn.nodemanager.resource.memory-mb는 원사 사용할 수 있습니다 얼마나 많은 자원을 말한다 (에서 반복 코멘트)

mapreduce 프로그램에서 해당 자원을 사용하려면 다음 매개 변수를 설정해야합니다.

mapreduce.map.memory.mb

mapreduce.map.java.opts

mapreduce.reduce.memory.mb

mapreduce.reduce.java.opts

java.opts를 memory.md보다 10-20 % 작게 설정하십시오.