oozie의 병렬 작업에 대해 별도로 오류 정리를 처리하는 방법

Question

oozie에서 oozie의 fork 옵션을 사용하여 실행할 수있는 병렬 작업 세트를 실행해야합니다. 이제 제가 직면하고있는 문제는 하나의 작업이 실패 할 경우 나머지 작업도 실패합니다. 왜냐하면 모든 단일 작업에 대해 오류가 발생하면 강제 종료 제어 노드를 호출하기 때문입니다. 웹에서 많이 검색했지만 모든 작업에 대해 별도로 오류를 처리하는 방법을 찾지 못했습니다.

도움을 주시면 감사하겠습니다. 다음과 같이

내 workflow.xml은 다음과 같습니다

<workflow-app name="WorkFlowForSshAction" xmlns="uri:oozie:workflow:0.1"> 
<start to="copyfroms3tohdfs"/> 
<action name="copyfroms3tohdfs"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${s3tohdfsscript}</command> 
<capture-output/> 
</ssh> 
<ok to="createhivetables"/> 
<error to="killAction"/> 
</action> 


<action name="createhivetables"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${createhivetablesscript}</command> 
<capture-output/> 
</ssh> 
<ok to="gold__pos_denorm_trn_itm_offr"/> 
<error to="killAction"/> 
</action> 
<action name="gold__pos_denorm_trn_itm_offr"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${denormalizationscript}</command> 
<capture-output/> 
</ssh> 
<ok to="forknode"/> 
<error to="killAction"/> 
</action> 
<fork name="forknode"> 
     <path start="gold__dypt_pos_trn_offr"/> 
     <path start="gold__hr_pos_trn_offr"/> 
       <path start="approach3"/> 
       <path start="aproach11"/> 
       <path start="aproach12"/> 
       <path start="aproach13"/> 
       <path start="aproach14"/> 
       <path start="aproach15"/> 
       <path start="aproach16"/> 
       <path start="aproach17"/> 

</fork> 
<action name="gold__dypt_pos_trn_offr"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${daypartscript}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="gold__hr_pos_trn_offr"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${hourscript}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="approach3"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach3script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach11"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach11script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach12"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach12script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach13"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach13script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach14"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach14script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach15"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach15script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach16"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach16script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<action name="aproach17"> 
<ssh xmlns="uri:oozie:ssh-action:0.1"> 
<host>${CMNodeLogin}</host> 
<command>${approach17script}</command> 
<capture-output/> 
</ssh> 
<ok to="joinnode"/> 
<error to="killAction"/> 
</action> 
<join name="joinnode" to="end"/> 
<kill name="killAction"> 
<message>"Killed job due to error"</message> 
</kill> 
<end name="end"/> 
</workflow-app> 

Answer 1

당신을위한 정리 작업을 수행하는 새로운 노드 (주로 자바)를 만듭니다. 또한 모든 "오류"조치를이 새 노드로 라우트하십시오. EL 함수를 사용하여 실제로 오류를 일으킨 노드를 식별 할 수 있습니다. $ {wf : lastErrorNode()}. 이를 클린업 처리 노드의 인수로 전달하면 Java 내부에서 정리할 논리 (java hdfs API 사용)를 수행 할 수 있습니다.

새로운 노드는 것 같은 :

<action name="myCleanUpAction"> 
<java> 
     <job-tracker>${jobTracker}</job-tracker> 
     <name-node>${nameNode}</name-node> 
     <main-class>com.foo.CleanUpMain</main-class> 
     <arg>${wf:lastErrorNode()}</arg> 
     <arg>any useful argument1</arg> 
     <arg>any useful argument2</arg> 
    </java> 
    <ok to="fail"/> 
    <error to="fail"/> 
</action>