[jira] [Created] (FLINK-16645) Limit the maximum backlogs in subpartitions for data skew case

[Shang Yuanchun (Jira)]

Zhijiang created FLINK-16645:
--------------------------------

             Summary: Limit the maximum backlogs in subpartitions for data skew case
                 Key: FLINK-16645
                 URL: https://issues.apache.org/jira/browse/FLINK-16645
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Network
            Reporter: Zhijiang
             Fix For: 1.11.0


In the case of data skew, most of the buffers in partition's LocalBufferPool are probably requested away and accumulated in certain subpartition, which would increase in-flight data to slow down the barrier alignment.

We can set up a proper config to control how many backlogs are allowed for one subpartition. If one subpartition reaches this threshold, it will make the buffer pool unavailable which blocks task processing continuously. Then we can reduce the in-flight data for speeding up checkpoint process a bit and not impact on the performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)