Hadoop distcp 명령이 작동하지 않습니다.

-1

안녕하세요, CDH4.3을 사용하는 클러스터에서 CDH4.5를 사용하는 클러스터로 데이터를 이동하려고합니다. 다음 명령을 실행하고 있습니다. 나는 다음과 같은 오류를 얻고 실행 한 후Hadoop distcp 명령이 작동하지 않습니다.

hadoop distcp -update hftp://server1:50070/hbase/test/x hdfs://server2:8020/copy/

14/01/28 19:42:43 INFO tools.DistCp: srcPaths=[hftp://server1:50070/hbase/test/x] 
14/01/28 19:42:43 INFO tools.DistCp: destPath=hdfs://server2:8020/copy 
14/01/28 19:42:45 INFO tools.DistCp: sourcePathsCount=1 
14/01/28 19:42:45 INFO tools.DistCp: filesToCopyCount=1 
14/01/28 19:42:45 INFO tools.DistCp: bytesToCopyCount=1 
14/01/28 19:42:46 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 
14/01/28 19:42:47 INFO mapred.JobClient: Running job: job_201401101918_0008 
14/01/28 19:42:48 INFO mapred.JobClient: map 0% reduce 0% 
14/01/28 19:43:05 INFO mapred.JobClient: map 100% reduce 0% 
14/01/28 19:43:07 INFO mapred.JobClient: Task Id : attempt_201401101918_0008_m_000000_0, Status : FAILED 
14/01/28 19:43:08 INFO mapred.JobClient: map 0% reduce 0% 
14/01/28 19:43:19 INFO mapred.JobClient: map 100% reduce 0% 
14/01/28 19:43:22 INFO mapred.JobClient: Task Id : attempt_201401101918_0008_m_000000_1, Status : FAILED 
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 
     at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582) 
     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) 
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) 
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) 
     at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:396) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) 
     at org.apache.hadoop.mapred.Child.main(Child.java:262) 

14/01/28 19:43:23 INFO mapred.JobClient: map 0% reduce 0% 
14/01/28 19:43:33 INFO mapred.JobClient: map 100% reduce 0% 
14/01/28 19:43:35 INFO mapred.JobClient: Task Id : attempt_201401101918_0008_m_000000_2, Status : FAILED 
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 
     at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582) 
     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) 
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) 
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) 
     at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:396) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) 
     at org.apache.hadoop.mapred.Child.main(Child.java:262) 

14/01/28 19:43:36 INFO mapred.JobClient: map 0% reduce 0% 
14/01/28 19:43:46 INFO mapred.JobClient: map 100% reduce 0% 
14/01/28 19:43:50 INFO mapred.JobClient: map 0% reduce 0% 
14/01/28 19:43:53 INFO mapred.JobClient: Job complete: job_201401101918_0008 
14/01/28 19:43:53 INFO mapred.JobClient: Counters: 6 
14/01/28 19:43:53 INFO mapred.JobClient: Job Counters 
14/01/28 19:43:53 INFO mapred.JobClient:  Failed map tasks=1 
14/01/28 19:43:53 INFO mapred.JobClient:  Launched map tasks=4 
14/01/28 19:43:53 INFO mapred.JobClient:  Total time spent by all maps in occupied slots (ms)=64095 
14/01/28 19:43:53 INFO mapred.JobClient:  Total time spent by all reduces in occupied slots (ms)=0 
14/01/28 19:43:53 INFO mapred.JobClient:  Total time spent by all maps waiting after reserving slots (ms)=0 
14/01/28 19:43:53 INFO mapred.JobClient:  Total time spent by all reduces waiting after reserving slots (ms)=0 
14/01/28 19:43:53 INFO mapred.JobClient: Job Failed: NA 
With failures, global counters are inaccurate; consider running with -i 
Copy failed: java.io.IOException: Job failed! 
     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1388) 
     at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667) 
     at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) 
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) 
     at org.apache.hadoop.tools.DistCp.main(DistCp.java:908) 

You have new mail in /var/spool/mail/root 
[[email protected] root]$ hadoop distcp -update hftp://server1:50070/hbase/test/x hdfs://server2:8020/copy hadoop distcp -update hftp://server1:50070/hbase/test/x hdfs://server2:8020/copy 
14/01/28 19:46:09 INFO tools.DistCp: srcPaths=[hftp://server1:50070/hbase/test/x, hdfs://server2:8020/copy, hadoop, distcp, hftp://server1:50070/hbase/test/x] 
14/01/28 19:46:09 INFO tools.DistCp: destPath=hdfs://server2:8020/copy 
With failures, global counters are inaccurate; consider running with -i 
Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source hadoop does not exist. 
Input source distcp does not exist. 
     at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:641) 
     at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656) 
     at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) 
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) 
     at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

내가 잘못 가고 어디 나를 인도 해주십시오.

출처

2014-01-28 piyush pankaj

는 지금

hadoop distcp -update hdfs://server1:8020/hbase/test/x hdfs://server2:8020/copy/

에 대한 해결책을 가지고 그러나 definatly HFTP 나를 위해 작동하지 않는 이유를 알고 싶습니다.

출처

2014-01-29 10:46:06

-1

나는 hftp에 대해 잘못된 포트 번호가 있다고 생각합니다. 50070은 namenode web ui의 기본 포트입니다.

시도 :

hadoop distcp -update hftp://server1/hbase/test/x hdfs://server2:8020/copy/

출처

2014-01-29 03:43:06 pensz

안녕하세요. Pensz 답장을 보내 주셔서 감사합니다. 그냥 당신이 제안한 것을 시도했지만 문제는 여전히 동일합니다. 나는 여전히 같은 오류가 발생합니다. –

시도 hadoop dfs -ls hftp : // server1/hbase/test/x. 맞습니까? – pensz

찾았습니다 -rw-r-r-- 3 루트 hbase 1 2014-01-28 18:57 hftp : // server1/hbase/test/x –

Hadoop distcp 명령이 작동하지 않습니다.

답변

관련 문제