Stephan Ewen created FLINK-1101:
-----------------------------------
Summary: Make memory management adaptive
Key: FLINK-1101
URL:
https://issues.apache.org/jira/browse/FLINK-1101 Project: Flink
Issue Type: Improvement
Components: Local Runtime
Affects Versions: 0.7-incubating
Reporter: Stephan Ewen
Assignee: Stephan Ewen
I suggest to rework the memory management.
Right now, it works the following way: When the program is submitted, it is checked how many memory consuming operations happen (sort, hash, explicit cache, ... ) Each one is assigned a static relative memory fraction, which the taskmanager provides.
This is a very conservative planning and mostly due to the fact that with the streaming runtime, we may have all operations running concurrently. But in fact we mostly have not and are therefore wasting memory by being too conservative.
To make the most of the available memory, I suggest to make the management adaptive:
- Operators need to be able to request memory bit by bit
- Operators need to be able to release memory on request. The sorter / hash table / cache do this naturally by spilling.
- Memory has to be redistributed between operators when new requesters come.
This also plays nicely with the idea of leaving all non-assigned memory to intermediate results, to allow for maximum caching of historic intermediate results.
This would solve [FLINK-170] and [FLINK-84].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Locked
