[jira] [Created] (FLINK-11869) [checkpoint] Make buffer size in checkpoint stream factory configurable

[Shang Yuanchun (Jira)]

Yun Tang created FLINK-11869:
--------------------------------

             Summary: [checkpoint] Make buffer size in checkpoint stream factory configurable
                 Key: FLINK-11869
                 URL: https://issues.apache.org/jira/browse/FLINK-11869
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Checkpointing
            Reporter: Yun Tang
            Assignee: Yun Tang
             Fix For: 1.9.0


Currently, the default buffer size for {{FsCheckpointStateOutputStream}} is only 4KB. This would case a lot of IOPS if stream is large. Unfortunately, when user want to checkpoint on a totally disaggregated file system which has no data node manager running in local machine, they might have a IOPS limit or cannot serve too many IOPS at a time. This would cause the checkpoint duration really large and might expire often.
If we want to increase this buffer size, we have to increase the {{fileStateThreshold}} to indirectly increase the buffer size. However, as we all know, too many not-so-small {{ByteStreamStateHandle}} returned to checkpoint coordinator would easily cause job manager OOM and checkpoint meta file large.

We should also make the buffer size configurable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)