[jira] [Created] (FLINK-11552) Akka association issues in 1.7.x

[Shang Yuanchun (Jira)]

William Cummings created FLINK-11552:
----------------------------------------

             Summary: Akka association issues in 1.7.x
                 Key: FLINK-11552
                 URL: https://issues.apache.org/jira/browse/FLINK-11552
             Project: Flink
          Issue Type: Bug
          Components: Cluster Management
    Affects Versions: 1.7.1, 1.7.0
            Reporter: William Cummings


When testing our application on 1.7.0 and 1.7.1, taskmanagers associate correctly, but when a job is submitted it enters the RUNNING state, but no work is ever done. In the jobmanager logs (w/ akka logging turned up to DEBUG & "akka.log.lifecycle.events: true") I can observe some akka errors. Eventually a taskmanager is lost, and the task fails.

Please let me know if there is any additional information I can collect to help diagnose. If someone can point me in the right direction I'd be happy to implement a fix.

I've attached the relevant logs below:
{noformat}
019-02-07 17:45:58,543 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:45:58,548 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:45:58,563 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:46:10,538 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:46:10,548 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:46:10,568 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:46:24,204 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:46:24,210 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:46:24,211 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:46:35,304 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:46:35,309 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:46:35,317 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:46:48,152 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:46:48,178 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:46:48,180 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:47:00,615 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:47:00,634 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:47:00,635 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:47:20,366 WARN  org.apache.flink.runtime.rest.handler.legacy.metrics.MetricFetcher  - Requesting TaskManager's path for query services failed.
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#1335681479]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
    at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
    at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
    at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
    at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
    at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
    at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
    at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
    at java.lang.Thread.run(Thread.java:745)
2019-02-07 17:47:20,367 INFO  akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef   - Message [akka.actor.Status$Failure] from Actor[akka://flink/deadLetters] to Actor[akka://flink/deadLetters] was not delivered. [38] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:47:20,367 INFO  akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef   - Message [akka.actor.Status$Failure] from Actor[akka://flink/deadLetters] to Actor[akka://flink/deadLetters] was not delivered. [39] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:47:21,058 INFO  akka.actor.EmptyLocalActorRef                                 - Message [org.apache.flink.types.SerializableOptional] from Actor[akka://flink/deadLetters] to Actor[akka://flink/temp/$8b] was not delivered. [40] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:47:26,910 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:47:26,927 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:47:26,952 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:47:36,616 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440]
2019-02-07 17:47:36,617 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79]
2019-02-07 17:47:36,617 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed]
2019-02-07 17:47:49,921 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:47:49,933 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:47:49,949 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:48:05,517 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:48:05,520 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:48:05,520 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:48:16,266 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:48:16,273 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:48:16,280 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:48:26,316 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:48:26,317 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:48:26,342 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:48:36,736 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:48:36,739 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:48:36,771 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:48:48,014 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:48:48,018 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:48:48,038 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:48:57,129 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed]
2019-02-07 17:48:57,129 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440]
2019-02-07 17:48:57,129 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79]
2019-02-07 17:49:12,894 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:49:12,899 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:49:13,005 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:49:25,532 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:49:25,532 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:49:25,559 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:49:39,784 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-065199ce40199b440:39271]] Caused by: [app-flink-taskmanager-065199ce40199b440: unknown error]
2019-02-07 17:49:39,786 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-034a6a653e17966ed:46051]] Caused by: [app-flink-taskmanager-034a6a653e17966ed: unknown error]
2019-02-07 17:49:39,790 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink-metrics@app-flink-taskmanager-0056cc1c18d1cff79:41833]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:49:54,111 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122] has failed, address is now gated for [50] ms. Reason: [Disassociated]
2019-02-07 17:49:54,976 WARN  org.apache.flink.runtime.rest.handler.legacy.metrics.MetricFetcher  - Requesting TaskManager's path for query services failed.
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#1335681479]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
    at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
    at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
    at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
    at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
    at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
    at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
    at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
    at java.lang.Thread.run(Thread.java:745)
2019-02-07 17:49:54,977 INFO  akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef   - Message [akka.actor.Status$Failure] from Actor[akka://flink/deadLetters] to Actor[akka://flink/deadLetters] was not delivered. [41] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:49:54,977 INFO  akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef   - Message [akka.actor.Status$Failure] from Actor[akka://flink/deadLetters] to Actor[akka://flink/deadLetters] was not delivered. [42] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:49:55,037 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79: unknown error]
2019-02-07 17:49:55,037 INFO  akka.remote.RemoteActorRef                                    - Message [org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation] from Actor[akka://flink/temp/$Mc] to Actor[akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122/user/taskmanager_0#1564625337] was not delivered. [43] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:49:55,668 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122]] Caused by: [app-flink-taskmanager-0056cc1c18d1cff79]
2019-02-07 17:49:55,669 INFO  akka.remote.RemoteActorRef                                    - Message [org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation] from Actor[akka://flink/deadLetters] to Actor[akka.tcp://flink@app-flink-taskmanager-0056cc1c18d1cff79:6122/user/taskmanager_0#1564625337] was not delivered. [44] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2019-02-07 17:49:57,343 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Item-DedupEvents-0 -> Item-TransformEventMapper-0 -> CheckUniqueMapper-0 -> CustomerToItemFlatMapper-0 (5/6) (684b98348707f04b1c1270874cd1d02c) switched from RUNNING to FAILED.
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connection unexpectedly closed by remote task manager 'app-flink-taskmanager-0056cc1c18d1cff79/10.100.23.120:6125'. This might indicate that the remote task manager was lost.
    at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:136)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377)
    at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at java.lang.Thread.run(Thread.java:745)
2019-02-07 17:49:57,343 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job Mode-Item-AppJob (be9764bbcb7f535b0d0c8bc767c8f651) switched from state RUNNING to FAILING.
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connection unexpectedly closed by remote task manager 'app-flink-taskmanager-0056cc1c18d1cff79/10.100.23.120:6125'. This might indicate that the remote task manager was lost.
    at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:136)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377)
    at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947)
    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at java.lang.Thread.run(Thread.java:745)
{noformat}
 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)