[jira] [Created] (FLINK-15140) Shuffle data compression does not work with BroadcastRecordWriter.

[Shang Yuanchun (Jira)]

Yingjie Cao created FLINK-15140:
-----------------------------------

             Summary: Shuffle data compression does not work with BroadcastRecordWriter.
                 Key: FLINK-15140
                 URL: https://issues.apache.org/jira/browse/FLINK-15140
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Network
    Affects Versions: 1.10.0
            Reporter: Yingjie Cao
             Fix For: 1.10.0


I tested the newest code of master branch last weekend with more test cases. Unfortunately, several problems were encountered, including a bug of compression.

When BroadcastRecordWriter is used, for pipelined mode, because the compressor copies the data back to the input buffer, however, the underlying buffer is shared when BroadcastRecordWriter is used. So we can not copy the compressed buffer back to the input buffer if the underlying buffer is shared. For blocking mode, we wrongly recycle the buffer when buffer is not compressed, and the problem is also triggered when BroadcastRecordWriter is used.

To fix the problem, for blocking shuffle, the reference counter should be maintained correctly, for pipelined shuffle, the simplest way maybe disable compression is the underlying buffer is shared. I will open a PR to fix the problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)