파이썬을 사용하여 하이브 udf에서 런타임 오류가 발생했습니다.

Question

직원의 경험을 계산하기 위해 파이썬에서 사용자 정의 하이브 udf를 작성하려고하는데 런타임 오류가 발생했습니다.이 문제를 해결하십시오. 이,

select transform(eno,ename,hiredate) 
using 'python empexp.py' 
as (eno,ename,hiredate,cdate,e) 
from emp; 

하지만 런타임 오류를 얻고있다가, 여기에 오류 로그입니다

import sys 
from datetime import datetime 
from dateutil.relativedelta import relativedelta 

cdate = datetime.now().date() 

for line in sys.stdin: 
    line = line.strip() 
    eno,ename,hiredate = line.split('\t') 

    hdate = datetime.strptime(hiredate,'%Y-%m-%d').date() 

    year = str(relativedelta(current_date,hdate).years) 
    month = str(relativedelta(current_date,hdate).months) 
    day = str(relativedelta(current_date,hdate).days) 

    e = year+'-'+month+'-'+day 
    list1 = [str(eno),str(ename),str(hiredate),str(cdate),str(e)] 

    print '\t'.join(list1) 

empexp.py 내가 하이브 쿼리를 제공하고 다음에 파이썬 스크립트입니다

Diagnostic Messages for this Task: 
Error: java.lang.RuntimeException: Hive Runtime Error while closing operators 
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207) 
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script. 
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585) 
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697) 
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697) 
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697) 
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189) 
... 8 more 


FAILED: Execution Error, return code 20003 from 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask. 
An error occurred when trying to close the Operator running your custom  

script. 

도와주세요. 미리 감사드립니다.

Answer 1

스크립트에서 함수를 호출하기 전에 파일을 분산 캐시에 추가해야합니다.

add file /path/empexp.py; 

다음 당신의 도움을

select transform(eno,ename,hiredate) 
using 'python empexp.py' 
as (eno,ename,hiredate,cdate,e) 
from emp;