[jira] [Created] (FLINK-22663) Release YARN resource very slow when cancel the job after some NodeManagers shutdown

[Shang Yuanchun (Jira)]

Jinhong Liu created FLINK-22663:
-----------------------------------

             Summary: Release YARN resource very slow when cancel the job after some NodeManagers shutdown
                 Key: FLINK-22663
                 URL: https://issues.apache.org/jira/browse/FLINK-22663
             Project: Flink
          Issue Type: Bug
          Components: Deployment / YARN
    Affects Versions: 1.12.2
            Reporter: Jinhong Liu


When I test flink on YARN, there is a case that may cause some problems.

Hadoop Version: 2.7.3

Flink Version: 1.12.2

I deploy a flink job on YARN, when the job is running I stop one NodeManager, after one or two minutes, the job is auto recovered. But in this situation, if I cancel the job, the containers cannot be released immediately, there are still some containers that are running include the app master. About 5 minutes later, these containers exit, and about 10 minutes later the app master exit.

I check the log of app master, seems it try to stop the containers on the NodeManger which I have already stopped.
{code:java}
2021-05-14 06:15:17,389 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job class tv.freewheel.reporting.fastlane.Fastlane$ (da883ab39a7a82e4d45a3803bc77dd6f) switched from state CANCELLING to CANCELED.
2021-05-14 06:15:17,389 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Stopping checkpoint coordinator for job da883ab39a7a82e4d45a3803bc77dd6f.
2021-05-14 06:15:17,390 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down
2021-05-14 06:15:17,408 INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher           [] - Job da883ab39a7a82e4d45a3803bc77dd6f reached globally terminal state CANCELED.
2021-05-14 06:15:17,409 INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher           [] - Shutting down cluster with state CANCELED, jobCancelled: true, executionMode: DETACHED
2021-05-14 06:15:17,409 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting YarnJobClusterEntrypoint down with application status CANCELED. Diagnostics null.
2021-05-14 06:15:17,409 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint.
2021-05-14 06:15:17,420 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Stopping the JobMaster for job class tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f).
2021-05-14 06:15:17,422 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing cache directory /tmp/flink-web-af72a00c-0ddd-4e5e-a62c-8244d6caa552/flink-web-ui
2021-05-14 06:15:17,432 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://ip-10-23-19-197.ec2.internal:43811 lost leadership
2021-05-14 06:15:17,432 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down complete.
2021-05-14 06:15:17,436 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Shut down cluster because application is in CANCELED, diagnostics null.
2021-05-14 06:15:17,436 INFO  org.apache.flink.yarn.YarnResourceManagerDriver              [] - Unregister application from the YARN Resource Manager with final status KILLED.
2021-05-14 06:15:17,458 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl     [] - Suspending SlotPool.
2021-05-14 06:15:17,458 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Close ResourceManager connection 493862ba148679a4f16f7de5ffaef665: Stopping JobMaster for job class tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f)..
2021-05-14 06:15:17,458 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl     [] - Stopping SlotPool.
2021-05-14 06:15:17,482 INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl        [] - Waiting for application to be successfully unregistered.
2021-05-14 06:15:17,566 INFO  org.apache.flink.runtime.history.FsJobArchivist              [] - Job da883ab39a7a82e4d45a3803bc77dd6f has been archived at hdfs:/realtime/flink-archive/da883ab39a7a82e4d45a3803bc77dd6f.
2021-05-14 06:15:17,589 INFO  org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
2021-05-14 06:15:17,590 INFO  org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] - Stopping JobDispatcherLeaderProcess.
2021-05-14 06:15:17,590 INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher           [] - Stopping dispatcher akka.tcp://[hidden email]:40340/user/rpc/dispatcher_1.
2021-05-14 06:15:17,590 INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher           [] - Stopping all currently running jobs of dispatcher akka.tcp://[hidden email]:40340/user/rpc/dispatcher_1.
2021-05-14 06:15:17,591 INFO  org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator [] - Shutting down back pressure request coordinator.
2021-05-14 06:15:17,591 INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher           [] - Stopped dispatcher akka.tcp://[hidden email]:40340/user/rpc/dispatcher_1.
2021-05-14 06:15:17,594 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Disconnect job manager [hidden email]://[hidden email]:40340/user/rpc/jobmanager_2 for job da883ab39a7a82e4d45a3803bc77dd6f from the resource manager.
2021-05-14 06:15:17,600 INFO  org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl [] - Interrupted while waiting for queue
java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_161]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_161]
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[?:1.8.0_161]
        at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287) [flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
2021-05-14 06:15:17,648 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:15:17,699 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
2021-05-14 06:15:17,741 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:15:17,796 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:15:17,809 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:15:17,813 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
2021-05-14 06:15:17,817 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:15:17,822 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:15:17,879 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:15:17,889 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
2021-05-14 06:15:17,898 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:15:17,903 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:15:17,907 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:15:17,911 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
2021-05-14 06:15:17,960 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
2021-05-14 06:15:17,964 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:43814] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:17,964 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:38826] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,016 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:45022] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,016 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:40808] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,061 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:33912] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,061 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:44652] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,120 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:42454] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,120 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:41756] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,125 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:36652] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,125 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:37709] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,126 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:40308] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,126 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:34524] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,141 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:44435] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,141 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:37224] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,143 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:38940] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,143 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:33014] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,202 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:39939] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,202 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:35165] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,204 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:45913] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,204 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:36333] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,220 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:35366] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,220 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:45411] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,223 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:34759] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,223 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:42621] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,228 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:40782] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,228 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:36612] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,251 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:38342] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:15:18,251 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:36176] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:18:18,220 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl          [] - Failed to stop Container container_1620970870707_0001_01_000057when stopping NMClientImpl
2021-05-14 06:18:18,240 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:18:18,295 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:18:18,299 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
2021-05-14 06:18:18,557 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:44399] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:18:18,557 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:33246] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:18:18,611 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:39100] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:18:18,611 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:35428] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:01,684 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:41510] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:01,730 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:39595] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:01,741 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:46788] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:01,754 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:46748] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:01,754 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:34218] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:01,761 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://[hidden email]:42730] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2021-05-14 06:21:18,522 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl          [] - Failed to stop Container container_1620970870707_0001_01_000078when stopping NMClientImpl
2021-05-14 06:21:18,567 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
2021-05-14 06:21:18,571 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
2021-05-14 06:21:18,605 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:21:18,657 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:21:18,698 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:21:18,702 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:21:18,705 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
2021-05-14 06:21:18,707 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
2021-05-14 06:24:18,934 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl          [] - Failed to stop Container container_1620970870707_0001_01_000008when stopping NMClientImpl
2021-05-14 06:24:18,986 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:24:19,036 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:24:19,077 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:24:19,080 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:24:19,083 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
2021-05-14 06:27:19,303 ERROR org.apache.hadoop.yarn.client.api.impl.NMClientImpl          [] - Failed to stop Container container_1620970870707_0001_01_000029when stopping NMClientImpl
2021-05-14 06:27:19,349 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
2021-05-14 06:27:19,353 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-28-67.ec2.internal:38916
2021-05-14 06:27:19,402 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:27:19,466 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:27:19,470 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:27:19,504 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:27:19,508 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-27-242.ec2.internal:41435
2021-05-14 06:27:19,510 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
2021-05-14 06:27:19,545 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:27:19,548 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-71.ec2.internal:44804
2021-05-14 06:27:19,551 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-86.ec2.internal:45099
2021-05-14 06:27:19,554 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-24-139.ec2.internal:42226
2021-05-14 06:27:19,557 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-19-197.ec2.internal:44443
2021-05-14 06:27:19,559 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy [] - Opening proxy : ip-10-23-25-241.ec2.internal:42723
2021-05-14 06:27:50,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2021-05-14 06:27:50,794 INFO  org.apache.flink.runtime.blob.BlobServer                     [] - Stopped BLOB server at 0.0.0.0:44447
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)