매우 느리게 실행되는 Hadoop mapreduce

xenserver에 vms로 설치된 1.1.2 버전의 4datanode/1namenode hadoop 클러스터를 사용하고 있습니다. 1GB의 텍스트 파일을 가지고 있으며 단어 개수를 시도했습니다. 지도는 2 시간이 걸렸으며 감속기가 멈추었습니다. 보통 펄 스크립트가 10 분 안에 작업을 마쳤습니다. 내 설정에서 뭔가 빠져있는 것처럼 보입니다.매우 느리게 실행되는 Hadoop mapreduce

Kbs의 작은 파일에도 거의 시간이 걸리지 않았습니다.

[email protected] ~]$ hadoop jar /usr/share/hadoop/hadoop-examples-1.1.2.jar wordcount huge out 
13/05/29 10:45:09 INFO input.FileInputFormat: Total input paths to process : 1 
13/05/29 10:45:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library 
13/05/29 10:45:09 WARN snappy.LoadSnappy: Snappy native library not loaded 
13/05/29 10:45:11 INFO mapred.JobClient: Running job: job_201305290801_0002 
13/05/29 10:45:12 INFO mapred.JobClient: map 0% reduce 0% 
13/05/29 10:57:14 INFO mapred.JobClient: map 2% reduce 0% 
13/05/29 10:58:01 INFO mapred.JobClient: map 3% reduce 0% 
13/05/29 10:58:53 INFO mapred.JobClient: map 4% reduce 0% 
13/05/29 10:58:54 INFO mapred.JobClient: map 5% reduce 0% 
13/05/29 10:59:33 INFO mapred.JobClient: map 6% reduce 0% 
13/05/29 11:01:52 INFO mapred.JobClient: map 7% reduce 0% 
13/05/29 11:03:02 INFO mapred.JobClient: map 8% reduce 0% 
13/05/29 11:03:20 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_0, Status : FAILED 
Task attempt_201305290801_0002_m_000002_0 failed to report status for 604 seconds. Killing! 
13/05/29 11:03:28 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000003_0, Status : FAILED 
Task attempt_201305290801_0002_m_000003_0 failed to report status for 604 seconds. Killing! 
13/05/29 11:03:29 INFO mapred.JobClient: map 9% reduce 0% 
13/05/29 11:04:07 INFO mapred.JobClient: map 10% reduce 0% 
13/05/29 11:05:13 INFO mapred.JobClient: map 11% reduce 0% 
13/05/29 11:06:34 INFO mapred.JobClient: map 12% reduce 0% 
13/05/29 11:06:59 INFO mapred.JobClient: map 13% reduce 0% 
13/05/29 11:08:14 INFO mapred.JobClient: map 14% reduce 0% 
13/05/29 11:08:39 INFO mapred.JobClient: map 15% reduce 0% 
13/05/29 11:09:35 INFO mapred.JobClient: map 16% reduce 0% 
13/05/29 11:10:03 INFO mapred.JobClient: map 17% reduce 0% 
13/05/29 11:10:55 INFO mapred.JobClient: map 18% reduce 0% 
13/05/29 11:11:47 INFO mapred.JobClient: map 19% reduce 0% 
13/05/29 11:14:05 INFO mapred.JobClient: map 20% reduce 0% 
13/05/29 11:15:22 INFO mapred.JobClient: map 21% reduce 0% 
13/05/29 11:15:49 INFO mapred.JobClient: map 22% reduce 0% 
13/05/29 11:17:09 INFO mapred.JobClient: map 23% reduce 0% 
13/05/29 11:18:06 INFO mapred.JobClient: map 24% reduce 0% 
13/05/29 11:18:29 INFO mapred.JobClient: map 25% reduce 0% 
13/05/29 11:18:53 INFO mapred.JobClient: map 26% reduce 0% 
13/05/29 11:20:05 INFO mapred.JobClient: map 27% reduce 0% 
13/05/29 11:21:09 INFO mapred.JobClient: map 28% reduce 0% 
13/05/29 11:21:45 INFO mapred.JobClient: map 29% reduce 0% 
13/05/29 11:22:14 INFO mapred.JobClient: map 30% reduce 0% 
13/05/29 11:22:31 INFO mapred.JobClient: map 31% reduce 0% 
13/05/29 11:22:32 INFO mapred.JobClient: map 32% reduce 0% 
13/05/29 11:23:01 INFO mapred.JobClient: map 33% reduce 0% 
13/05/29 11:23:41 INFO mapred.JobClient: map 34% reduce 0% 
13/05/29 11:24:29 INFO mapred.JobClient: map 35% reduce 0% 
13/05/29 11:25:16 INFO mapred.JobClient: map 36% reduce 0% 
13/05/29 11:25:58 INFO mapred.JobClient: map 37% reduce 0% 
13/05/29 11:27:09 INFO mapred.JobClient: map 38% reduce 0% 
13/05/29 11:27:55 INFO mapred.JobClient: map 39% reduce 0% 
13/05/29 11:28:33 INFO mapred.JobClient: map 40% reduce 0% 
13/05/29 11:29:50 INFO mapred.JobClient: map 41% reduce 0% 
13/05/29 11:30:29 INFO mapred.JobClient: map 42% reduce 0% 
13/05/29 11:31:37 INFO mapred.JobClient: map 43% reduce 0% 
13/05/29 11:32:10 INFO mapred.JobClient: map 44% reduce 0% 
13/05/29 11:32:34 INFO mapred.JobClient: map 45% reduce 0% 
13/05/29 11:34:08 INFO mapred.JobClient: map 46% reduce 0% 
13/05/29 11:36:01 INFO mapred.JobClient: map 47% reduce 0% 
13/05/29 11:36:57 INFO mapred.JobClient: map 48% reduce 0% 
13/05/29 11:37:53 INFO mapred.JobClient: map 49% reduce 0% 
13/05/29 11:39:50 INFO mapred.JobClient: map 50% reduce 0% 
13/05/29 11:42:17 INFO mapred.JobClient: map 51% reduce 0% 
13/05/29 11:43:26 INFO mapred.JobClient: map 52% reduce 0% 
13/05/29 11:47:55 INFO mapred.JobClient: map 53% reduce 0% 
13/05/29 11:48:25 INFO mapred.JobClient: map 54% reduce 0% 
13/05/29 11:49:28 INFO mapred.JobClient: map 54% reduce 2% 
13/05/29 11:49:31 INFO mapred.JobClient: map 54% reduce 4% 
13/05/29 11:50:03 INFO mapred.JobClient: map 55% reduce 4% 
13/05/29 11:50:49 INFO mapred.JobClient: map 56% reduce 4% 
13/05/29 11:50:54 INFO mapred.JobClient: map 58% reduce 4% 
13/05/29 11:51:21 INFO mapred.JobClient: map 59% reduce 4% 
13/05/29 11:51:46 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_1, Status : FAILED 
Task attempt_201305290801_0002_m_000002_1 failed to report status for 685 seconds. Killing! 
13/05/29 11:52:09 INFO mapred.JobClient: map 61% reduce 4% 
13/05/29 11:52:27 INFO mapred.JobClient: map 62% reduce 4% 
13/05/29 11:52:53 INFO mapred.JobClient: map 63% reduce 4% 
13/05/29 11:53:36 INFO mapred.JobClient: map 64% reduce 4% 
13/05/29 11:53:57 INFO mapred.JobClient: map 65% reduce 4% 
13/05/29 11:54:41 INFO mapred.JobClient: map 66% reduce 4% 
13/05/29 11:55:51 INFO mapred.JobClient: map 67% reduce 4% 
13/05/29 11:57:00 INFO mapred.JobClient: map 68% reduce 4% 
13/05/29 11:57:04 INFO mapred.JobClient: map 69% reduce 4% 
13/05/29 11:57:11 INFO mapred.JobClient: map 70% reduce 4% 
13/05/29 11:57:41 INFO mapred.JobClient: map 71% reduce 4% 
13/05/29 11:58:13 INFO mapred.JobClient: map 72% reduce 4% 
13/05/29 11:58:45 INFO mapred.JobClient: map 73% reduce 4% 
13/05/29 11:59:05 INFO mapred.JobClient: map 74% reduce 4% 
13/05/29 11:59:08 INFO mapred.JobClient: map 74% reduce 6% 
13/05/29 11:59:42 INFO mapred.JobClient: map 75% reduce 6% 
13/05/29 11:59:52 INFO mapred.JobClient: map 76% reduce 6% 
13/05/29 12:00:33 INFO mapred.JobClient: map 77% reduce 6% 
13/05/29 12:00:53 INFO mapred.JobClient: map 78% reduce 6% 
13/05/29 12:01:06 INFO mapred.JobClient: map 79% reduce 6% 
13/05/29 12:01:51 INFO mapred.JobClient: map 80% reduce 6% 
13/05/29 12:02:29 INFO mapred.JobClient: map 81% reduce 6% 
13/05/29 12:02:39 INFO mapred.JobClient: map 82% reduce 6% 
13/05/29 12:02:56 INFO mapred.JobClient: map 83% reduce 6% 
13/05/29 12:03:36 INFO mapred.JobClient: map 84% reduce 6% 
13/05/29 12:04:05 INFO mapred.JobClient: map 85% reduce 6% 
13/05/29 12:04:59 INFO mapred.JobClient: map 86% reduce 6% 
13/05/29 12:05:47 INFO mapred.JobClient: map 87% reduce 6% 
13/05/29 12:07:04 INFO mapred.JobClient: map 88% reduce 6% 
13/05/29 12:08:00 INFO mapred.JobClient: map 89% reduce 6% 
13/05/29 12:08:32 INFO mapred.JobClient: map 90% reduce 6% 
13/05/29 12:09:41 INFO mapred.JobClient: map 91% reduce 6% 
13/05/29 12:10:04 INFO mapred.JobClient: map 92% reduce 6% 
13/05/29 12:10:17 INFO mapred.JobClient: map 93% reduce 6% 
13/05/29 12:10:45 INFO mapred.JobClient: map 94% reduce 6% 
13/05/29 12:10:49 INFO mapred.JobClient: map 95% reduce 6% 
13/05/29 12:11:00 INFO mapred.JobClient: map 96% reduce 6% 
13/05/29 12:11:03 INFO mapred.JobClient: map 97% reduce 6% 
13/05/29 12:11:12 INFO mapred.JobClient: map 98% reduce 6% 
13/05/29 12:11:17 INFO mapred.JobClient: map 99% reduce 6% 
13/05/29 12:12:02 INFO mapred.JobClient: map 100% reduce 6% 






^C[[email protected] ~]$

출처

2013-05-29 user2433024

어쩌면 스와핑이 진행되고있는 것처럼 보입니다. 그러나 클러스터에 너무 국한되어 있으므로 아무 것도 될 수는 없습니다. –

은 매퍼를 공유하고 코드를 줄이면 텍스트 출력이 문제를 진단하는 데 도움이되지 않습니다. – Woot4Moo

제공 한 제한된 정보 (콘솔 출력)에서 클러스터가 건강하지 않은 것처럼 보입니다.

13/05/29 11:03:20 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_0, Status : FAILED 
Task attempt_201305290801_0002_m_000002_0 failed to report status for 604 seconds. Killing!

10 분 안에 JobTracker에보고하지 않은 노드에서 작업을 시도했습니다. 이로 인해 작업이 다시 예약되었습니다. 더 많은 로그로 들어가면 할당 된 작업에 실패한 특정 노드를 식별하는 작업이 필요할 수 있습니다.

출처

2013-05-29 16:26:53

CDH를 설치하여 해결했습니다. – user2433024

매우 느리게 실행되는 Hadoop mapreduce

답변

관련 문제