[jira] [Created] (FLINK-22754) Finished job not always archived properly

[Shang Yuanchun (Jira)]

Mathieu DESPRIEE created FLINK-22754:
----------------------------------------

             Summary: Finished job not always archived properly
                 Key: FLINK-22754
                 URL: https://issues.apache.org/jira/browse/FLINK-22754
             Project: Flink
          Issue Type: Bug
    Affects Versions: 1.12.1
            Reporter: Mathieu DESPRIEE


Flink batch job, ran on YARN/EMR. Flink v1.12.1

2 different runs, same configuration.

At the end of the first run, the job has been properly archived:
{quote}2021-05-20 07:14:39,479 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Shut down cluster because application is in SUCCEEDED, diagnostics null.
 2021-05-20 07:14:39,479 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Unregister application from the YARN Resource Manager with final status SUCCEEDED.
 2021-05-20 07:14:39,490 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl [] - Waiting for application to be successfully unregistered.
 2021-05-20 07:14:39,501 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 2021-05-20 07:14:40,277 INFO org.apache.hadoop.conf.Configuration.deprecation [] - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
 2021-05-20 07:14:40,531 WARN com.amazonaws.services.s3.internal.Mimetypes [] - Unable to find 'mime.types' file in classpath
 2021-05-20 07:14:40,727 INFO org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
 2021-05-20 07:14:40,728 INFO org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] - Stopping JobDispatcherLeaderProcess.
 2021-05-20 07:14:40,728 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping dispatcher akka.tcp://[hidden email]:44219/user/rpc/dispatcher_1.
 2021-05-20 07:14:40,728 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping all currently running jobs of dispatcher akka.tcp://[hidden email]:44219/user/rpc/dispatcher_1.
 2021-05-20 07:14:40,729 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator [] - Shutting down back pressure request coordinator.
 2021-05-20 07:14:40,729 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopped dispatcher akka.tcp://[hidden email]:44219/user/rpc/dispatcher_1.
 2021-05-20 07:14:40,815 INFO *org.apache.flink.runtime.history.FsJobArchivist [] - Job a2760da3fb41127b8d6f54a1573bd3fa has been archived at s3://REDACTED-emr/history-server/a2760da3fb41127b8d6f54a1573bd3fa*.
{quote}
 

At the end of the second run, no mention of "FsJobArchivist"
{quote}2021-05-21 23:49:00,191 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Shut down cluster because application is in SUCCEEDED, diagnostics null.
2021-05-21 23:49:00,193 INFO org.apache.flink.yarn.YarnResourceManagerDriver [] - Unregister application from the YARN Resource Manager with final status SUCCEEDED.
2021-05-21 23:49:00,208 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl [] - Waiting for application to be successfully unregistered.
2021-05-21 23:49:01,311 INFO org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] - Stopping JobDispatcherLeaderProcess.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping dispatcher akka.tcp://[hidden email]:35331/user/rpc/dispatcher_1.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping all currently running jobs of dispatcher akka.tcp://[hidden email]:35331/user/rpc/dispatcher_1.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator [] - Shutting down back pressure request coordinator.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopped dispatcher akka.tcp://[hidden email]:35331/user/rpc/dispatcher_1.
2021-05-21 23:49:01,337 WARN com.amazonaws.services.s3.internal.Mimetypes [] - Unable to find 'mime.types' file in classpath
2021-05-21 23:49:01,460 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Closing the SlotManager.
2021-05-21 23:49:01,460 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Suspending the SlotManager.
2021-05-21 23:49:01,462 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:40643
2021-05-21 23:49:01,464 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
2021-05-21 23:49:01,487 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
{quote}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)