Flink "allow lateness" for tables

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink "allow lateness" for tables

Ramya Ramamurthy
Hi,

I would like to know if there is some configuration which enabled to
configure allow lateness in table. The documentation mentions about streams
and not tables.
If this is not present, is there a way to collect it on a side output for
tables.

Today, we see some late packet drops in Flink, where my tumbling window is
of 1 second and check-pointing every 5 mins.

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Flink "allow lateness" for tables

Jark Wu-2
Hi Ramya,

Are you looking for "allow lateness" for tumbling window in Flink SQL?

If yes, Flink SQL doesn't provide such things currently.
However, blink planner (the new runner of Flink SQL) will support this
maybe in the next release (v1.10).
Actually, we already implemented it in blink planner but didn't expose the
feature in v1.9 to let us have enough time to review the API.

For now, you can convert Table to DataStream, and can use tumbling window
functionality on the DataStream which supports allow lateness.
Or you can increase the watermark delay to wait more late records, but the
window output delay will get larger.

Best,
Jark

On Wed, 17 Jul 2019 at 17:39, Ramya Ramamurthy <[hidden email]> wrote:

> Hi,
>
> I would like to know if there is some configuration which enabled to
> configure allow lateness in table. The documentation mentions about streams
> and not tables.
> If this is not present, is there a way to collect it on a side output for
> tables.
>
> Today, we see some late packet drops in Flink, where my tumbling window is
> of 1 second and check-pointing every 5 mins.
>
> Thanks.
>