지형 공간 데이터가있는 테이블 (예 : 모양 컬럼)에 대한 파티션을 만드는 데 문제가 있습니다. 지형 공간 데이터를 가지고있는 테이블 빌딩이 있는데 나는 statename을 기반으로 파티션을 만들고 싶습니다. 하이브의 지형 공간 데이터 파티셔닝
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=non-strict;
create table buildingpartition (OBJECTID string,
> RILUNIQUEID string,
> RILFEATURECODE string,
> BLDGNO string,
> BLDGNAME string,
> BLDGTYPE string,
> BLDGSUBTYPE string,
> BLDGCLASS string,
> BLDGROAD string,
> BLDGSUBROAD string,
> SUBLOCALITY string,
> BLDGSIZE string,
> TAG string,
> PINCODE string,
> NUMBEROFFLATS string,
> NUMBEROFSHOPS string,
> BLDG_TYPE string,
> CABLEOPERATORNAME string,
> AREA_1 string,
> LBU2 string,
> SHAPE binary)
> Partitioned by (STATENAME string)
> ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde'
> STORED AS INPUTFORMAT 'com.esri.json.hadoop.UnenclosedJsonInputFormat'
> OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> Insert into table buildingpartition parition (statename) select * from building;
나는에 의한
로 오류의 주요 원인을 얻고있다 org.apache.hadoop.hive.ql.metadata합니다. HiveException : org.apache.hadoop.ipc.RemoteException (org.apache.hadoop.hdfs.protocol.FSLimitException $ PathComponentTooLongException) : statename = 00의 최대 경로 구성 요소 이름 제한 00 00 00 03 05 00 00 00 83 54 0b b0 17 22 41 f0 6d 9e 04 7b 61 43 41 c0 84 25 eb ed 17 22 41 a0 82 cd c2 80 61 43 41 01 00 00 05 00 00 00 00 00 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 40 a0 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 in 디렉토리 /apps/hive/warehouse/proddb.db/buildingpartition/.hive-staging_hive_2016-12-15_10-58-15_294_7141760028369054723-1/ _task_tmp.-ext-10002가 초과되었습니다. 제한 = 255 길이 = 408
누구든지이 문제를 해결할 수 있습니까? 아래에 전체 오류 추적을 첨부했습니다. 어떤 도움을 주시면 감사하겠습니다. 감사합니다
오류 : java.lang.RuntimeException가 : org.apache.hadoop.hive.ql.metadata.HiveException : 하이브 런타임 오류 행 { "OBJECTID"처리하는 동안 : 널 (null)에게, "riluniqueid": "KOL_A_LO0000_0597215를", " "bldgclass": "Building", "bldgroad": "bldgclass": "bldgclass": " : "PURANO BAZAR BRIDGE", "bldgsubroad": "PK VISHWAS ROAD", "sublocality": "Milan Garh Colony", "statename": "West Bengal", "bldgsize": "G + 0", "tag" "null", "pincode": "700119", "numberofflats": "0", "numberofshops": "0", "bldg_type": "기타", "cableoperatorname": "null", "area_1": "0.0 ","lbu2 ":"null ","shape ": T "A m {aCA % "A a T "AIXz aCA @ "A aCA % "Ap M {aCA @ 1r "A m {aCA T "AIXz aCA} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map (ExecMapper. java : 172) at org.apache.hadoop.mapred.MapRunner.run (MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper (MapTask.java:453) at org.apache. hadoop.mapred.MapTask.run (MapTask.java:343) at org.apache.hadoop.mapred.YarnChild $ 2.run (YarnChild.java:168) at java.security.AccessController.doPrivileged (기본 메소드) at org.apache.hadoop.mapred.YarnChild.main에서 org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1709) 에서 javax.security.auth.Subject.doAs (Subject.java:422) (YarnChild.java:162) 의해 발생 : 행 { "OBJECTID"을 처리하는 동안 하이브 런타임 오류 : org.apache.hadoop.hive.ql.metadata.HiveException NULL, "riluniqueid": "KOL_A_LO0000_0597215", "rilfeaturecode": "LO0000", "bldgno" : "null", "bldgname": "null", "bldgtype": "1", "bldgsubtype": "분류되지 않음", "bldgclass": "건물", "bldgroad": "PURANO BAZAR BRIDGE", "bldgsubroad" : "PK VISHWAS ROAD", "sublocality": "Milan Garh Colony", "statename": "West Bengal", "bldgsize": "G + 0", "tag": "null", "pincode": "700119 ","numberofflats ":"0 ","numberofshops ":"0 ","bldg_type ":"기타 ","cableoperatorname ":"널 (null) ","에리어 1 ":"0.0 ","lbu2 ":"null ","shape ": T "A m {aCA % "A a T "AIXz aCA @ "A aCA % "Ap M {aCA @ 1r "A m {aCA T "AIXz aCA} at org.apache.hadoop.hive.ql .exec.MapOperator.process (MapOperator.java:562) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map (ExecMapper.java:163) ... 8 더보기 발생 원인 : org.apache.hadoop.hive.ql.metadata.HiveException : org.apache.hadoop.hive.ql.metadata.HiveException : org.apache.hadoop.ipc.RemoteException (org.apache.hadoop.hdfs.protocol.FSLimitException $ PathComponentTooLongException) : statename = 00의 최대 경로 구성 요소 이름 제한 00 00 00 03 05 00 0 0 00 00 83 54 0b b0 17 22 41 f0 6d 9e 04 7b 61 43 41 c0 84 25 eb ed 17 22 41 a0 82 cd c2 80 61 43 41 01 00 00 00 05 00 00 00 00 00 00 83 53 0b b0 17 22 41 18 49 58 7a 80 61 43 41 40 40 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 in 디렉토리 /apps/hive/warehouse/proddb.db/buildingpartition/.hive-staging_hive_2016-12-15_10- 5815_294_7141760028369054723-1/_task_tmp.-ext-10002가 초과되었습니다. limit = 255 length = 408 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxComponentLength (FSDirectory.java:949) at org. (FSDirMkdirOp.java:237)org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createChildrenDirectories에서 org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createSingleDirectory (FSDirMkdirOp.java:191) (FSDirMkdirOp.java:166) 에서 691,363,210에서 org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createAncestorDirectories (FSDirMkdirOp.java:134) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal (FSNamesystem.java:2615) at org. apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt (FSNamesystem.java:2493) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile (FSNamesystem.java:2377) at org.apache. hadoop.hdfs.server.namenode.NameNodeRpcServer.create (NameNodeRpcServer.java:708) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTr org.apache.hadoop.ipc.ProtobufRpcEngine $ 서버 $ ProtoBufRpcInvoker에서 anslatorPB.create org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ ClientNamenodeProtocol $ 2.callBlockingMethod (ClientNamenodeProtocolProtos.java)에서 (ClientNamenodeProtocolServerSideTranslatorPB.java:405) .call (ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC $ Server.call (RPC.java:969) at org.apache.hadoop.ipc.Server $ Handler $ 1.run (Server. java : 2206) at org.apache.hadoop.ipc.Server $ Handler $ 1.run (Server.java:2202) at java.security.AccessController.doPrivileged (네이티브 메소드) (at javax.security.auth.Subject). doAs (Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server $ Handler.run (Server.java:2200) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles (FileSinkOperator.java:577) at org. apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths (FileSinkOperator.java:859) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths (FileSinkOperator.java:966) at org.apache. hadoop.hive.ql.exec.FileSinkOperator.process (FileSinkOperator.java:709) at org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:838) at org.apache.hadoop. hive.ql.exec.SelectOperator.process (SelectOperator.java:88) at org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:838) at org.apache.hadoop.hive. ql.exec.TableScanOperator.process (TableScanOperator.java:117) at orgapache.hadoop.hive.ql.exec.MapOperator $ MapOpCtx.forward (MapOperator.java:167) at org.apache.hadoop.hive.ql.exec.MapOperator.process (MapOperator.java:552) ... 9 more 발생 원인 : org.apache.hadoop.hive.ql.metadata.HiveException : org.apache.hadoop.ipc.RemoteException (org.apache.hadoop.hdfs.protocol.FSLimitException $ PathComponentTooLongException) : 최대 경로 구성 요소 이름 statename = 00 00 00 03 05 00 00 00 83 54 0b b0 17 22 41 f0 6d 9e 04 7b 61 43 41 c0 84 25 eb ed 17 22 41 a0 82 cd c2 80 61 43 41 01 00 00 00 05 00 00 00 00 00 00 00 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 40 a0 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 디렉토리의 /apps/hive/warehouse/proddb.db/ buildingpartition/.hive-staging_hive_2016-12-15_10-58-15_294_ 7141760028369054723-1/_task_tmp.-ext-10002가 초과되었습니다. limit = 255 length = 408 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxComponentLength (FSDirectory.java:949) at org.apache.hadoop .hdfs.server.namenode.FSDirectory.FSDirectory.addLastINode (FSDirectory.java:1014) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.unprotectedMkdir (FSDirMkdirOp.java:237) at org.apache.hadoop.hdfs .server.namenode.FSDirMkdirOp.createSingleDirectory (FSDirMkdirOp.java:191) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createChildrenDirectories (FSDirMkdirOp.java:166) at org.apache.hadoop.hdfs.server .namenode.FSDirMkdirOp.createAncestorDirectories (FSDirMkdirOp.java:134) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal (FSNamesystem.java : (FSNamesystem.java : 2493) 에서 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile 조직에서 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create (ClientNamenodeProtocolServerSideTranslatorPB.java:405) 에서 org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create (NameNodeRpcServer.java:708) 에서 . org.apache에서 org.apache.hadoop.ipc.ProtobufRpcEngine $ 서버 $ ProtoBufRpcInvoker.call (ProtobufRpcEngine.java:616) 에서 apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ ClientNamenodeProtocol $ 2.callBlockingMethod (ClientNamenodeProtocolProtos.java) .hadoop.ipc.RPC $ Server.call (RPC.java:969) 에서 org.apache.hadoop.ipc.Server $ Handler $ 1.run (Server.java:2206) at org.apache.hadoop.ipc.Server $ Handler $ 1.run (Server.java:2202) at java. security.AccessController.doPrivileged (네이티브 메소드) at javax.security.auth.Subject.doAs (Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1709) at org .apache.hadoop.ipc.Server $ Handler.run (Server.java:2200) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter (HiveFileFormatUtils.java:249) at org.apache.hadoop .hive.ql.exec.FileSinkOperator.createBucketForFileIdx (FileSinkOperator.java:622) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles (FileSinkOperator.java:566) ... 18 more 발생 원인 : org.apache.hadoop.ipc.RemoteException (org.apache.hadoop.hdfs.protocol.FSLimitException $ PathComponentTooLongException) : statename = 00의 최대 경로 구성 요소 이름 제한 00 00 00 03 05 00 00 00 00 83 54 0b b0 17 22 41 f0 6d 9e 04 7b 61 43 41 c0 84 25 eb ed 17 22 41 a0 82 cd c2 80 61 43 41 01 00 00 00 05 00 00 00 00 00 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 40 40 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 in 디렉토리 /apps/hive/warehouse/proddb.db/buildingpartition/.hive-staging_hive_2016-12-15_10- 58-15_294_7141760028369054723-1/_task_tmp.-ext-10002가 초과되었습니다. limit = 255 length = 408 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxComponentLength (FSDirectory.java:949) at org.apache . 하둡.hdfs.server.namenode.FSDirectory.addLastINode (FSDirectory.java:1014) org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.unprotectedMkdir (FSDirMkdirOp.java:237)에서 org.apache.hadoop.hdfs에서 . server.namenode.FSDirMkdirOp.createSingleDirectory org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createChildrenDirectories에서 (FSDirMkdirOp.java:191) (FSDirMkdirOp.java:166) org.apache.hadoop.hdfs.server에서 . namenode.FSDirMkdirOp.createAncestorDirectories (FSDirMkdirOp.java:134) org.apache.hadoop.hdfs.server.namenode에서 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal (FSNamesystem.java:2615)에서 . org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile에서 FSNamesystem.startFileInt (FSNamesystem.java:2493) (FSNamesystem.java:2377) 012 351 641 061 조직에서 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create (ClientNamenodeProtocolServerSideTranslatorPB.java:405) 에서 org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create (NameNodeRpcServer.java:708) 에서. org.apache에서 org.apache.hadoop.ipc.ProtobufRpcEngine $ 서버 $ ProtoBufRpcInvoker.call (ProtobufRpcEngine.java:616) 에서 apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ ClientNamenodeProtocol $ 2.callBlockingMethod (ClientNamenodeProtocolProtos.java) org.apache.hadoop.ipc.Server $ 처리기 $의 1.run (Server.java:2206) org.apache.hadoop.ipc에서 에서 .hadoop.ipc.RPC $ Server.call (RPC.java:969) . 서버 핸들러 $의 $의 1.run java.security.AccessController.doPrivileged (기본 방법)에서 (Server.java:2202) $ org.apache.hadoop.ipc.Server에서 org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1709)에서 javax.security.auth.Subject.doAs (Subject.java:422) 에서 16,org.apache.hadoop.ipc.Client.call에서 org.apache.hadoop.ipc.Client.call (Client.java:1426) 에서 Handler.run (Server.java:2200) (Client.java:1363) org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke (ProtobufRpcEngine.java:229) org.apache.hadoop.hdfs.protocolPB에서 com.sun.proxy. $ Proxy14.create (알 수없는 소스)에서 에서 . sun.reflect.NativeMethodAc에서 sun.reflect.NativeMethodAccessorImpl.invoke0 (기본 방법)에서 ClientNamenodeProtocolTranslatorPB.create (ClientNamenodeProtocolTranslatorPB.java:298) cessorImpl.invoke (NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)에서 java.lang.reflect.Method.invoke (Method.java:498)에서 org.apache에서 . com.sun.proxy에서 hadoop.io.retry.RetryInvocationHandler.invokeMethod (RetryInvocationHandler.java:256) org.apache.hadoop.io.retry.RetryInvocationHandler.invoke (RetryInvocationHandler.java:104)에서 . $ Proxy15.create org.apache.hadoop.hdfs.DFSClient.create에서 org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate (DFSOutputStream.java:1716) (DFSClient.java:1703) org.apache에서 에서 (알 소스). org.apache.hadoop.hdfs.DistributedFileSys에서 hadoop.hdfs.DFSClient.create (DFSClient.java:1638) org.apache.hadoop.fs.FileSystemLinkResolver.resolve에서 org.apache.hadoop.hdfs.DistributedFileSystem $ 7.doCall (DistributedFileSystem.java:445) 에서이 $ 7.doCall (DistributedFileSystem.java:449) (FileSystemLinkResolver.java : 81)에서 org.apache.hadoop org.apache.hadoop.hdfs.DistributedFileSystem.create (DistributedFileSystem.java:388에서 org.apache.hadoop.hdfs.DistributedFileSystem.create (DistributedFileSystem.java:445) 에서 ) . org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter에서 org.apache.hadoop.fs.FileSystem.create (FileSystem.java:802) 에서 fs.FileSystem.create (FileSystem.java:909) (HiveIgnoreKeyTextOutputFormat.java:80) org.apache.hadoop.hive.ql.io.HiveFileFormatUtils에서 .getRecordWriter (HiveFileFormatUtils.java:261) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter (HiveFileFormatUtils.java:246) ... 20 more 컨테이너가 ApplicationMaster에 의해 종료되었습니다. 컨테이너 요청시 사망했습니다. 바로 거기 그것은 말한다 org.apache.hadoop.hive.ql.exec.mr.MapRedTask에서 실행 오류, 리턴 코드 2