[jira] [Created] (FLINK-12069) Add proper lifecycle management for intermediate result partitions

[Shang Yuanchun (Jira)]

Till Rohrmann created FLINK-12069:
-------------------------------------

             Summary: Add proper lifecycle management for intermediate result partitions
                 Key: FLINK-12069
                 URL: https://issues.apache.org/jira/browse/FLINK-12069
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Coordination, Runtime / Network
    Affects Versions: 1.8.0, 1.9.0
            Reporter: Till Rohrmann


In order to properly execute batch jobs, we should make the lifecycle management of intermediate result partitions the responsibility of the {{JobMaster}}/{{Scheduler}} component. The {{Scheduler}} knows best when an intermediate result partition is no longer needed and, thus, can be freed. So for example, a blocking intermediate result should only be released after all subsequent blocking intermediate results have been completed in order to speed up potential failovers.

Moreover, having explicit control over intermediate result partitions, could also enable use cases like result partition sharing between jobs and even across clusters (by simply not releasing the result partitions).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)