Apache Flink: aligning watermark among parallel tasks

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Flink: aligning watermark among parallel tasks

Ozan DENİZ
We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App.

During the streaming process, all the 4 tasks' watermark values must be close to trigger window event.





  For example;

 
  Task 1 watermark value = 8

 
  Task 2 watermark value = 1

 
  Task 3 watermark value = 8

 
  Task 4 watermark value = 8





Task 2 is waiting for log to update its watermark. However,
the condition can occur before Task 2's update and we want to fire the
window event before it.




Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?
     
Reply | Threaded
Open this post in threaded view
|

Apache Flink: aligning watermark among parallel tasks

Ozan DENİZ
We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App.

During the streaming process, all the 4 tasks' watermark values must be close to trigger window event.





  For example;

 
  Task 1 watermark value = 8

 
  Task 2 watermark value = 1

 
  Task 3 watermark value = 8

 
  Task 4 watermark value = 8





Task 2 is waiting for log to update its watermark. However,
the condition can occur before Task 2's update and we want to fire the
window event before it.




Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?
         
mxm
Reply | Threaded
Open this post in threaded view
|

Re: Apache Flink: aligning watermark among parallel tasks

mxm
Hi Ozan,

You probably want to look at a custom Trigger implementation. Please see
the different triggers in
org/apache/flink/streaming/api/windowing/triggers/. You can write your own
event-based trigger. Best thing would be to extend the EventTimeTrigger
with your logic.

Then you can use windowedStream.trigger(new MyCustomTrigger()) to include
your trigger logic on your windowed stream.

Cheers,
Max


On Thu, Mar 24, 2016 at 10:51 AM, Ozan DENİZ <[hidden email]> wrote:

> We are using periodic event time window with watermark. We have currently
> 4 parallel tasks in our Flink App.
>
> During the streaming process, all the 4 tasks' watermark values must be
> close to trigger window event.
>
>
>
>
>
>   For example;
>
>
>   Task 1 watermark value = 8
>
>
>   Task 2 watermark value = 1
>
>
>   Task 3 watermark value = 8
>
>
>   Task 4 watermark value = 8
>
>
>
>
>
> Task 2 is waiting for log to update its watermark. However,
> the condition can occur before Task 2's update and we want to fire the
> window event before it.
>
>
>
>
> Is there any mechanism to align all the parallel tasks' watermarks or fire
> the window event without waiting for other tasks?
>
>
>