Zhilong Hong created FLINK-21332:
------------------------------------
Summary: Optimize releasing result partitions in RegionPartitionReleaseStrategy
Key: FLINK-21332
URL:
https://issues.apache.org/jira/browse/FLINK-21332 Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: Zhilong Hong
Fix For: 1.13.0
RegionPartitionReleaseStrategy is responsible for releasing result partitions when all the downstream tasks finish.
The current implementation is:
{code:java}
for each consumed SchedulingResultPartition of current finished SchedulingPipelinedRegion:
for each consumer SchedulingPipelinedRegion of the SchedulingResultPartition:
if all the regions are finished:
release the partitions
{code}
The time complexity of releasing a result partition is O(N^2). However, considering that during the entire stage, all the result partitions need to be released, the time complexity is actually O(N^3).
After the optimization of DefaultSchedulingTopology, the consumed result partitions are grouped. Since the result partitions in one group are isomorphic, we can just cache the finished status of result partition groups and the corresponding pipeline regions.
The optimized implementation is:
{code:java}
for each ConsumedPartitionGroup of current finished SchedulingPipelinedRegion:
if all consumer SchedulingPipelinedRegion of the ConsumedPartitionGroup are finished:
set the ConsumePartitionGroup to be fully consumed
for result partition in the ConsumePartitionGroup:
if all the ConsumePartitionGroups it belongs to are fully consumed:
release the result partition
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
