Stephan Ewen created FLINK-19023:
------------------------------------
Summary: Remove pruning of Record Serializer Buffer
Key: FLINK-19023
URL:
https://issues.apache.org/jira/browse/FLINK-19023 Project: Flink
Issue Type: Sub-task
Components: Runtime / Network
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.12.0
Currently, the {{SpanningRecordSerializer}} prunes its internal serialization buffer under special circumstances:
- The buffer becomes larger than a certain threshold (5MB)
- The full record end lines up exactly with a full buffer length (this change got introduced at some point, it is not clear what the purpose is)
This optimization virtually never kicks in (because of the second condition) and also seems unnecessary. There is only a single serializer on the sender side, so this will not help to reduce the maximum memory footprint needed in any way.
NOTE: A similar optimization on the reader side ({{SpillingAdaptiveSpanningRecordDeserializer}}) makes sense, because multiple parallel deserializers run in order to piece together the records when retrieving buffers from the network in arbitrary order. Truncating buffers (or spilling) there helps reduce the maximum required memory footprint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
