zhijiang created FLINK-15187:
--------------------------------
Summary: Reuse LocalBufferPool for FileBufferReader in blocking partition
Key: FLINK-15187
URL:
https://issues.apache.org/jira/browse/FLINK-15187 Project: Flink
Issue Type: Task
Components: Runtime / Network
Reporter: zhijiang
If we take the file type via `taskmanager.network.bounded-blocking-subpartition-type` for batch job, while creating the respective view for reading the subpartition persistent data, it would create two unpolled memory segments for every subpartition. This portion of temporary memory is not managed and calculated by framework, so it might cause OOM error concern.
We can also reuse the ResultPartition's `LocalBufferPool` to read subpartition data to avoid this memory overhead. But there are additional two problems for reuse directly.
* The current core size of `LocalBufferPool` is `numberOfSubpartitions + 1`, but every subpartition needs two segments for pre-reading atm. We can remove the pre-reading to make the current core pool size suitable for the reading requirements, because the pre-reading function seems has no obvious benefits in practice which is only effecting for the last data.
* When task finishes, it would destroy the `LocalBufferPool` even though the respective `ResultPartition still alive, so the following subpartition view can not reuse the pool directly. We should adjust the respective logics to either delay destroy the pool or create a new pool for subpartition view.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
