AWS EMR S3DistCp : auxService : mapreduce_shuffle이 존재하지 않습니다.

Question

SSH를 통해 AWS EMR v5.4.0 인스턴스에 연결되었으며 s3distcp를 호출하려고합니다. This link는 설정에 EMR 단계를 호출하는 방법을 보여줍니다,하지만 난 그것을 실행할 때 나는 다음과 같은 오류가 발생합니다 :

Container launch failed for container_1492469375740_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) 
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) 
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155) 
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:390) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

내가 the instructions here을 따라하지만 여전히 작동하지 않았다.

Answer 1

내가 mapreduce_shuffle를 구성한 후에 restart the yarn nodemanager service에 필요한 밝혀 : /etc/hadoop/conf/yarn-site.xml :

$ initctl list | grep yarn 
hadoop-yarn-resourcemanager start/running, process 1256 
hadoop-yarn-proxyserver start/running, process 702 
hadoop-yarn-nodemanager start/running, process 896 
$ sudo stop hadoop-yarn-nodemanager 
$ sudo start hadoop-yarn-nodemanager 

또한, 경우에 그것은 yarn-site.xml 파일이 위치했다하는 데 도움이됩니다. 이미 yarn.nodemanager.aux-services에 대한 항목을했지만, mapreduce_shuffle 구성되지 않았습니다 :

<property> 
    <name>yarn.nodemanager.aux-services</name> 
    <value>spark_shuffle,mapreduce_shuffle</value> 
</property> 

<property> 
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> 
    <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property> 

<property> 
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name> 
    <value>org.apache.spark.network.yarn.YarnShuffleService</value> 
</property> 

:

<property> 
    <name>yarn.nodemanager.aux-services</name> 
    <value>spark_shuffle,</value> 
</property> 

<property> 
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name> 
    <value>org.apache.spark.network.yarn.YarnShuffleService</value> 
</property> 

그래서 내가 이런 식으로 추가