Using event timestamps

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using event timestamps

Gyula Fóra-2
Hey All!

Is there a proper way of using a Flink Streaming source with event
timestamps and watermarks? What I mean here is instead of implementing a
custom SourceFunction, use an existing one and provide some Timestamp
extractor (like the one currently used for Time windows), which will also
automatically generate the watermarks based on event time (assuming that
timestamps are partially ordered).

Maybe I missed something, or there might be somebody working on this
already :)

Cheers,
Gyula
Reply | Threaded
Open this post in threaded view
|

Re: Using event timestamps

Márton Balassi
Hey Gyula,

I have been recently looking at the streaming UdfOperators and can not
recall a utility for the sources that you are looking for, but maybe I am
also missing it. :)
It would be a convenient addition though.

Best,

Marton

On Sun, Sep 13, 2015 at 8:59 PM, Gyula Fóra <[hidden email]> wrote:

> Hey All!
>
> Is there a proper way of using a Flink Streaming source with event
> timestamps and watermarks? What I mean here is instead of implementing a
> custom SourceFunction, use an existing one and provide some Timestamp
> extractor (like the one currently used for Time windows), which will also
> automatically generate the watermarks based on event time (assuming that
> timestamps are partially ordered).
>
> Maybe I missed something, or there might be somebody working on this
> already :)
>
> Cheers,
> Gyula
>
Reply | Threaded
Open this post in threaded view
|

Re: Using event timestamps

Aljoscha Krettek-2
Hi,
there is nothing like that in there yet but it would be very nice to have
it. :D Also, as far as I know no-one is working on this right now.

Cheers,
Aljoscha

On Mon, 14 Sep 2015 at 06:27 Márton Balassi <[hidden email]>
wrote:

> Hey Gyula,
>
> I have been recently looking at the streaming UdfOperators and can not
> recall a utility for the sources that you are looking for, but maybe I am
> also missing it. :)
> It would be a convenient addition though.
>
> Best,
>
> Marton
>
> On Sun, Sep 13, 2015 at 8:59 PM, Gyula Fóra <[hidden email]> wrote:
>
> > Hey All!
> >
> > Is there a proper way of using a Flink Streaming source with event
> > timestamps and watermarks? What I mean here is instead of implementing a
> > custom SourceFunction, use an existing one and provide some Timestamp
> > extractor (like the one currently used for Time windows), which will also
> > automatically generate the watermarks based on event time (assuming that
> > timestamps are partially ordered).
> >
> > Maybe I missed something, or there might be somebody working on this
> > already :)
> >
> > Cheers,
> > Gyula
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Using event timestamps

Gyula Fóra
Then as a first step, I will open a Jira for it :)

Aljoscha Krettek <[hidden email]> ezt írta (időpont: 2015. szept. 14.,
H, 10:18):

> Hi,
> there is nothing like that in there yet but it would be very nice to have
> it. :D Also, as far as I know no-one is working on this right now.
>
> Cheers,
> Aljoscha
>
> On Mon, 14 Sep 2015 at 06:27 Márton Balassi <[hidden email]>
> wrote:
>
> > Hey Gyula,
> >
> > I have been recently looking at the streaming UdfOperators and can not
> > recall a utility for the sources that you are looking for, but maybe I am
> > also missing it. :)
> > It would be a convenient addition though.
> >
> > Best,
> >
> > Marton
> >
> > On Sun, Sep 13, 2015 at 8:59 PM, Gyula Fóra <[hidden email]> wrote:
> >
> > > Hey All!
> > >
> > > Is there a proper way of using a Flink Streaming source with event
> > > timestamps and watermarks? What I mean here is instead of implementing
> a
> > > custom SourceFunction, use an existing one and provide some Timestamp
> > > extractor (like the one currently used for Time windows), which will
> also
> > > automatically generate the watermarks based on event time (assuming
> that
> > > timestamps are partially ordered).
> > >
> > > Maybe I missed something, or there might be somebody working on this
> > > already :)
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>