Watermarking in Src and Timestamping downstream

[Roshan Naik-2]

Have streaming use cases where it is useful & easier to generate the watermark in the Source (via ctx.emitWatermark() ) and assign timestamps in a downstream custom operator which calls  output.collect(new StreamRecord<T>(msg, time)).
When doing so, I see that the watermark reaches the downstream operator, but does not flow past it and consequently further downstream windows remain open. (I am using Flink 1.6). 
-roshan

[Stephan Ewen]

Do you know what part of the code happens to block off your watermark?
Maybe a method that is overridden in AbstractStreamOperator in your code?

On Sat, Aug 10, 2019 at 4:06 AM Roshan Naik <[hidden email]>
wrote:

> Have streaming use cases where it is useful & easier to generate the
> watermark in the Source (via ctx.emitWatermark() ) and assign timestamps
> in a downstream custom operator which calls  output.collect(new
> StreamRecord<T>(msg, time)).
> When doing so, I see that the watermark reaches the downstream operator,
> but does not flow past it and consequently further downstream windows
> remain open. (I am using Flink 1.6).
> -roshan
>
>

[Roshan Naik-2]

 Thanks Stephan. It was the case,.. I had an empty override of the processWatermark() in the operator that went unnoticed. Removing it fixed the problem.-roshan
    On Monday, August 12, 2019, 02:39:45 AM PDT, Stephan Ewen <[hidden email]> wrote:  
 
 Do you know what part of the code happens to block off your watermark?
Maybe a method that is overridden in AbstractStreamOperator in your code?

On Sat, Aug 10, 2019 at 4:06 AM Roshan Naik <[hidden email]>
wrote:

> Have streaming use cases where it is useful & easier to generate the
> watermark in the Source (via ctx.emitWatermark() ) and assign timestamps
> in a downstream custom operator which calls  output.collect(new
> StreamRecord<T>(msg, time)).
> When doing so, I see that the watermark reaches the downstream operator,
> but does not flow past it and consequently further downstream windows
> remain open. (I am using Flink 1.6).
> -roshan
>
>