0
독립형 노드에서 용량 스케줄러 대기열을 설정하려고하는데 문제가 있습니다.용량 스케줄러 큐 문제
UI를 통해 Cloudera Manager에서 볼 수있는 대기열을 만들 수 있었지만 로그인하고 매핑 대기열 - 목록을 만들려고 할 때 대기열이 표시되지 않습니다. 단지 config에서 생략 한 기본 대기열 만 보여줍니다.
용량 스케줄러 구성 : 로그 아래
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>card,bank,digital</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.digital.capacity</name>
<value>41</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.card.capacity</name>
<value>29</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.bank.capacity</name>
<value>30</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.digital.acl_submit_applications</name>
<value>cloudera</value>
</property>
2014-02-18 08:09:54,940 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing card
capacity = 0.29 [= (float) configuredCapacity/100 ]
asboluteCapacity = 0.29 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 2900 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 2900 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ]
2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: card: capacity=0.29, absoluteCapacity=0.29, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing bank
capacity = 0.3 [= (float) configuredCapacity/100 ]
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ]
2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: bank: capacity=0.3, absoluteCapacity=0.3, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing digital
capacity = 0.41 [= (float) configuredCapacity/100 ]
asboluteCapacity = 0.41 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 4100 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 4100 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:cloudera [= configuredAcls ]
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: digital: capacity=0.41, absoluteCapacity=0.41, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: root: numChildQueue= 3, capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, asboluteMaxCapacity=1.0, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_APPLICATIONS:*
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing card
capacity = 0.29 [= (float) configuredCapacity/100 ]
asboluteCapacity = 0.29 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 2900 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 2900 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ]
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: card: capacity=0.29, absoluteCapacity=0.29, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing bank
capacity = 0.3 [= (float) configuredCapacity/100 ]
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS: [= configuredAcls ]
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: bank: capacity=0.3, absoluteCapacity=0.3, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing digital
capacity = 0.41 [= (float) configuredCapacity/100 ]
asboluteCapacity = 0.41 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity)/100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 4100 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 4100 [= (int)(maxApplications * (userLimit/100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory/minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit/100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory/(clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory/clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory)/maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:cloudera [= configuredAcls ]
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: digital: capacity=0.41, absoluteCapacity=0.41, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cloudera IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService RESULT=SUCCESS
2014-02-18 08:10:19,772 INFO org.apache.hadoop.yarn.server.resourcemanager.AdminService: RM Admin: refreshAdminAcls invoked by user cloudera
2014-02-18 08:10:19,792 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cloudera IP=127.0.0.1 OPERATION=refreshAdminAcls TARGET=AdminService RESULT=SUCCESS
Mapred 명령 출력
Queue Name : default
Queue State : running
Scheduling Info : Queue configuration
Capacity Percentage: 100.0%
User Limit: 100%
Priority Supported: NO
-------------
Map tasks
Capacity: 2 slots
Used capacity: 0 (0.0% of Capacity)
Running tasks: 0
-------------
Reduce tasks
Capacity: 2 slots
Used capacity: 0 (0.0% of Capacity)
Running tasks: 0
-------------
Job info
Number of Waiting Jobs: 0
Number of Initializing Jobs: 0
Number of users who have submitted jobs: 0
당신도 설정 yarn.resourcemanager.scheduler.class 매개 변수 org.apache.hadoop에게 한 사용자가 큐에 100 % 자원을 사용할 수 있습니다. conf/yarn-site.xml의 yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler? 변경 사항을 적용하려면 클러스터를 다시 시작해야 할 수도 있습니다. 다시 시작한 후에 대기열 목록을 수정 한 경우 "$ HADOOP_YARN_HOME/bin/yarn rmadmin -refreshQueues"를 실행하여 대기열 목록을 새로 고침해야합니다. – rVr
답장을 보내 주셔서 감사합니다. 나는 이것을 이미 시도했지만 여전히 효과가 없다. – user3315248
문제를 파악했습니다. 근본적인 원인은 클라이언트 구성이 배포되었지만/etc/hadoop/conf가 배포 된 최신 구성을 가리 키지 않았습니다. 나는 그것을 수동으로 바꿨고 효과가 있었다. – user3315248