Watermark at the end of job

[Roshan Naik-2]

When generating watermarks outside the source using a assignTimestampsAndWatermarks()  .. once the source has finished generating all the messages, I don't see evidence of a final watermark getting generated to flush windows. Do the windows auto flush on job termination without need for the final watermark ?
-roshan

[Jark Wu-2]

Hi Roshan,

I think the final watermark is generated by the source operator which wraps
user-defined source function and emit a final watermark when source
function finished [1].  And the watermark operator (which wraps your
assignTimestampsAndWatermarks) will forward only the final watermark [2].

[1]
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/StreamSource.java#L99
[2]
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/TimestampsAndPeriodicWatermarksOperator.java#L95

Regards,
Jark

On Sat, 23 Feb 2019 at 18:21, Roshan Naik <[hidden email]>
wrote:

> When generating watermarks outside the source using
> a assignTimestampsAndWatermarks()  .. once the source has finished
> generating all the messages, I don't see evidence of a final watermark
> getting generated to flush windows. Do the windows auto flush on job
> termination without need for the final watermark ?
> -roshan