java.lang.NoSuchMethodError : org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf

Question

나는 내가 BasicMRTest.java

내 pom.xml 파일을 실행하고이 link 에서 사용할 수있는 맵리 듀스 단위 테스트를 실행하는 것을 시도하고있다

<?xml version="1.0" encoding="UTF-8"?> 
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>cooltool</groupId> 
    <artifactId>tool</artifactId> 
    <version>1.0.0-SNAPSHOT</version> 
    <repositories> 
     <repository> 
      <id>cloudera</id> 
      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> 
     </repository> 
    </repositories> 
    <dependencies> 
     <!-- Dependencies that are shaded and relocated --> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-mapreduce-client-core</artifactId> 
      <version>2.7.2</version> 
      <type>jar</type> 
      <scope>provided</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-common</artifactId> 
      <version>2.7.2</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-test</artifactId> 
      <version>2.6.0-mr1-cdh5.7.0-SNAPSHOT</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-hdfs</artifactId> 
      <version>2.6.0-cdh5.7.0-SNAPSHOT</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-hdfs</artifactId> 
      <version>2.6.0-cdh5.5.0</version> 
      <classifier>tests</classifier> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-common</artifactId> 
      <version>2.6.0-cdh5.7.0-SNAPSHOT</version> 
      <classifier>tests</classifier> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-core</artifactId> 
      <version>2.6.0-mr1-cdh5.7.0-SNAPSHOT</version> 
     </dependency> 
     <!-- Dependencies not to include (related to tests) --> 
     <dependency> 
      <groupId>org.apache.mrunit</groupId> 
      <artifactId>mrunit</artifactId> 
      <version>0.9.0-incubating</version> 
      <classifier>hadoop2</classifier> 
     </dependency> 
     <dependency> 
      <groupId>junit</groupId> 
      <artifactId>junit</artifactId> 
      <version>4.11</version> 
      <scope>test</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.mockito</groupId> 
      <artifactId>mockito-core</artifactId> 
      <version>1.10.19</version> 
      <scope>test</scope> 
     </dependency> 
     <dependency> 
      <groupId>com.github.tomakehurst</groupId> 
      <artifactId>wiremock</artifactId> 
      <version>1.55</version> 
      <scope>test</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.powermock</groupId> 
      <artifactId>powermock-core</artifactId> 
      <version>1.6.4</version> 
      <scope>test</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.powermock</groupId> 
      <artifactId>powermock-module-junit4</artifactId> 
      <version>1.6.4</version> 
      <scope>test</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.powermock</groupId> 
      <artifactId>powermock-api-mockito</artifactId> 
      <version>1.6.4</version> 
      <scope>test</scope> 
     </dependency> 
     <dependency> 
      <groupId>com.beust</groupId> 
      <artifactId>jcommander</artifactId> 
      <version>1.48</version> 
     </dependency> 
    </dependencies> 
    <build> 
     <plugins> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-compiler-plugin</artifactId> 
       <version>3.5.1</version> 
       <configuration> 
        <source>1.7</source> 
        <target>1.7</target> 
       </configuration> 
      </plugin> 
     </plugins> 
    </build> 
</project> 

하지만 오류가 나는

Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.101 sec <<< FAILURE! 
testClusterWithData(com.amazon.ws.emr.hadoop.fs.migration.BasicMRTest) Time elapsed: 1.067 sec <<< ERROR! 
java.lang.NoSuchMethodError: org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(Ljava/lang/String;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/htrace/core/HTraceConfiguration; 
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:822) 
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) 
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) 
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1107) 
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:978) 
    at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:807) 
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:467) 
    at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:426) 
    at com.amazon.ws.emr.hadoop.fs.migration.BasicMRTest.setUp(BasicMRTest.java:51) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) 
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) 
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) 
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) 
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) 
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) 
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) 
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) 
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) 
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) 
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) 
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309) 
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) 
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) 
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) 
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) 
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) 
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) 
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) 
testClusterWithData(com.amazon.ws.emr.hadoop.fs.migration.BasicMRTest) Time elapsed: 1.068 sec <<< ERROR! 
java.lang.NullPointerException 
    at com.amazon.ws.emr.hadoop.fs.migration.BasicMRTest.tearDown(BasicMRTest.java:60) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) 
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) 
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) 
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) 
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) 
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) 
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) 
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) 
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) 
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) 
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) 
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309) 
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) 
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) 
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) 
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) 
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) 
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) 
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) 

이 사람이 좀 도와 주시겠습니까

은 - 수 오류? 다른 버전의 hadoop을 시도했지만 여전히 동일한 오류가 발생합니다.

Answer 1

나는 똑같은 문제에 직면하고 내 발견을 공유하여 누군가를 도울 수 있도록했다.

데이터를 HDFS에 쓰는 Talend 작업이 있으므로 한 구성 요소에서 Hadoop 배포 유형을 선택해야합니다. Hadoop은 cloudera이지만 구성 요소에는 Amazon EMR이 있습니다. 그래서이 오류가 발생했습니다. 배포를 cloudera로 변경하면 효과가있었습니다. 그래서 이것은 Jar 파일이 사용되는 라이브러리와 관련이 있다고 생각합니다.