Ufuk Celebi created FLINK-1833:
----------------------------------
Summary: Refactor partition availability notification in ExecutionGraph
Key: FLINK-1833
URL:
https://issues.apache.org/jira/browse/FLINK-1833 Project: Flink
Issue Type: Improvement
Components: Distributed Runtime, JobManager
Affects Versions: master
Reporter: Ufuk Celebi
The mechanism to notify the JobManager about available result partitions is hard to understand. The are two parts to this:
1) JobManager
- The deployment of receivers happens in the Execution class although it is by now totally unrelated to the state of a specific execution. I propose to move this to the respective IntermediateResultPartition.
- The deployment information for a receiver is spread across different components: when creating the TaskDeploymentDescriptor and the "caching" of partition infos at the consuming vertex. This is very hard to follow and results in unnecessary messages being sent (which are discarded at the TM).
2) TaskManager
- Pipelined results notify where you would expect it in the ResultPartition, but blocking results don't have an extra message and are implicitly piggy-backed to the final state transition, after which the job manager deploys receivers if all blocking partitions of a result have been produced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Locked
