[jira] [Created] (FLINK-18405) Add watermark support for unaligned checkpoints

[Shang Yuanchun (Jira)]

Arvid Heise created FLINK-18405:
-----------------------------------

             Summary: Add watermark support for unaligned checkpoints
                 Key: FLINK-18405
                 URL: https://issues.apache.org/jira/browse/FLINK-18405
             Project: Flink
          Issue Type: Improvement
    Affects Versions: 1.12.0
            Reporter: Arvid Heise


Currently, Flink generates the watermark as a first step of recovery instead of
storing the latest watermark in the operators to ease rescaling. In unaligned
checkpoints, that means on recovery, Flink generates watermarks after it
restores in-flight data. If your pipeline uses an operator that applies the
latest watermark on each record, it will produce incorrect results during
recovery if the watermark is not directly or indirectly part of the operator
state. Thus, SQL OVER operator should not be used with unaligned
checkpoints, while window operators are safe to use.

A possible solution is to store the watermark in the operator state. If rescaling may occur, watermarks should be stored per key-group in a union-state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)