hadoop 파일이 저장된 노드

Question

특정 hdfs 파일이 저장된 datanode 또는 hdfs 파일을 저장하는 블록 목록을 찾는 방법이 있습니까? 내가 HDFS 경우

예를 들어, : //user/person/file.csv을

는 데이타 노드에서이 파일을 구성하는 블록에 해당하는 ext4를 경로의 목록을 찾을 수있는 방법이 있습니까?

Answer 1

예, HDFS의 다른 데이터 노드에 저장된 블록의 위치를 ​​확인할 수 있습니다. "/user/hduser/file.txt"이 당신에게 파일을 만들어 각각의 블록에 관련된 모든 정보를 제공합니다

hdfs fsck /user/hduser/file.txt -files -blocks -locations 

: 여기에 명령입니다. 출력은 일반적으로 다음과 같습니다 : "REPL"후 정보

[hduser@node001 ~]$ hdfs fsck /user/hduser/file.txt -files -blocks -locations 
Connecting to namenode via http://node001.morado.com:50070 
FSCK started by hduser (auth:SIMPLE) from /192.168.2.169 for path /user/hduser/file.txt at Mon Jul 11 23:14:27 PDT 2016 
/user/hduser/file.txt 1073839694 bytes, 9 block(s): OK 
0. BP-778802867-192.168.2.147-1465886958278:blk_1080847742_7107323 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-25d2b73a-2dc2-48c1-9aad-f0f5ca8d302a,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-293a7f8d-ad31-4bc1-98d8-0c0822eda305,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-8efb7a6e-08f0-4f2d-aee2-bc5a102277bd,DISK]] 
1. BP-778802867-192.168.2.147-1465886958278:blk_1080847748_7107329 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-6881b609-1473-48d5-a07c-f111e0bdcf2f,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-060c75ff-5632-4f6f-a73b-fb2a68927c63,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-3e108776-d3bd-4b84-b68a-59e1ca755331,DISK]] 
2. BP-778802867-192.168.2.147-1465886958278:blk_1080847753_7107334 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-25d2b73a-2dc2-48c1-9aad-f0f5ca8d302a,DISK], DatanodeInfoWithStorage[192.168.2.177:50010,DS-b7a33931-8917-4fe2-b2ec-2e4c3d5b6b01,DISK], DatanodeInfoWithStorage[192.168.2.135:50010,DS-5efb0813-7e4e-4d27-8fa4-7f8f3b2e6e3c,DISK]] 
3. BP-778802867-192.168.2.147-1465886958278:blk_1080847760_7107341 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-6881b609-1473-48d5-a07c-f111e0bdcf2f,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-b8a5ceaf-6953-4842-8930-29b286ccb7cf,DISK], DatanodeInfoWithStorage[192.168.2.134:50010,DS-c6418fbb-6e30-447e-b507-bf19e0f28fd1,DISK]] 
4. BP-778802867-192.168.2.147-1465886958278:blk_1080847764_7107345 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-95636645-c59e-4bca-8478-c15b3c16d514,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-293a7f8d-ad31-4bc1-98d8-0c0822eda305,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-8efb7a6e-08f0-4f2d-aee2-bc5a102277bd,DISK]] 
5. BP-778802867-192.168.2.147-1465886958278:blk_1080847768_7107349 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-6881b609-1473-48d5-a07c-f111e0bdcf2f,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-276bd2c8-ee3d-4cd3-b655-17a83917c45b,DISK], DatanodeInfoWithStorage[192.168.2.134:50010,DS-5e128658-c876-46df-b10e-5962baf73db2,DISK]] 
6. BP-778802867-192.168.2.147-1465886958278:blk_1080847772_7107353 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-fa57d5e9-a187-4856-8bf2-6933e63b3afe,DISK], DatanodeInfoWithStorage[192.168.2.135:50010,DS-f5f1e2a0-186b-4c70-844f-7e7ebe389f50,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-8cfb8ffb-77b6-40bb-930e-81c7198166ad,DISK]] 
7. BP-778802867-192.168.2.147-1465886958278:blk_1080847776_7107357 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-95636645-c59e-4bca-8478-c15b3c16d514,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-060c75ff-5632-4f6f-a73b-fb2a68927c63,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-3e108776-d3bd-4b84-b68a-59e1ca755331,DISK]] 
8. BP-778802867-192.168.2.147-1465886958278:blk_1080847780_7107361 len=97870 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-25d2b73a-2dc2-48c1-9aad-f0f5ca8d302a,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-de3cffa6-cef4-4f47-9bbf-5f44214b3a5a,DISK], DatanodeInfoWithStorage[192.168.2.177:50010,DS-847ec520-bc14-4ca4-af94-21140a3b20f6,DISK]] 

Status: HEALTHY 
Total size: 1073839694 B 
Total dirs: 0 
Total files: 1 
Total symlinks:  0 
Total blocks (validated): 9 (avg. block size 119315521 B) 
Minimally replicated blocks: 9 (100.0 %) 
Over-replicated blocks: 0 (0.0 %) 
Under-replicated blocks: 0 (0.0 %) 
Mis-replicated blocks:  0 (0.0 %) 
Default replication factor: 3 
Average block replication: 3.0 
Corrupt blocks:  0 
Missing replicas:  0 (0.0 %) 
Number of data-nodes:  7 
Number of racks:  1 
FSCK ended at Mon Jul 11 23:14:27 PDT 2016 in 1 milliseconds 


The filesystem under path '/user/hduser/file.txt' is HEALTHY 

봐는, 일반적으로 "DatanodeInfoWithStorage"태그로 시작. 데이터 노드 위치에 대한 필수 정보를 제공합니다.