[DISCUSSION] Timely function interface and timer params

[Gyula Fóra-2]

Hello,

I was thinking about the methods provided by the timely functions and the
timerservice and I am wondering if it makes sense to change them a little
so they can cover a wider set of use case. Maybe I missed something
completely obvious so please shoot me down in that case :)

Currently the user gets a TimerService to register timers that will in the
future call the onTimer method. It is not completely obvious to me how
would I implement a function that needs to trigger two types of callbacks
in the future. If I get only one onTimer method I should be able to pass in
some sort of parameter or flag so I can branch in my onTimer
implementation.

As parameters are not supported I am left with states that are scoped to
the keys which is also pretty useless if I want to trigger different timed
actions for the same keys.

I know this is quite tricky but I see some alternative options:
 - The register timer method returns a unique (per key) timer id, so we can
associate state with this id to fetch info about the timer registered. (We
could also remove timers with this id and maybe add methods to remove all
for the current key)
 - Allow the users to pass a custom parameter when registering the
callback, and the parameter would be passed to the onTimer method
 - Allow users to pass custom callback functions when registering the
timers, but this would mean we have to support some sort of context for
accessing the state (like the window context before)
 - We could go for an annotation based API like in beam but thats probably
not good to mix in the current ones

I personally prefer the first one.

What do you think?

Regards,
Gyula

[Aljoscha Krettek-2]

Hi Gyula,
if you look at the internal API you'll notice that it is pretty much like
your second proposal. Just for reference, the interface is roughly this:

public interface InternalTimerService<N> {
  long currentProcessingTime();
  long currentWatermark();
  void registerProcessingTimeTimer(N namespace, long time);
  void deleteProcessingTimeTimer(N namespace, long time);
  void registerEventTimeTimer(N namespace, long time);
  void deleteEventTimeTimer(N namespace, long time);
}

that namespace bit can be anything for which you can provide a
TypeSerializer.

IMHO, this goes back a bit to the discussion about adding a low level
operator-like interface that allows pretty much anything a custom operator
can do but with a defined, stable interface. The internal operator
interface is somewhat in flux, still, so I wouldn't recommend people to use
it directly.

The only thing missing, really, from TimelyFlatMap is access to namespaces
for state and timers. With that, you could implement even the
WindowOperator as a TimelyFlatMap since I worked on abstracting everything
that it uses away behind interfaces that any operator can use. The last
pice, the generic timer API went in last, of course. :-)

Cheers,
Aljoscha

On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <[hidden email]> wrote:

> Hello,
>
> I was thinking about the methods provided by the timely functions and the
> timerservice and I am wondering if it makes sense to change them a little
> so they can cover a wider set of use case. Maybe I missed something
> completely obvious so please shoot me down in that case :)
>
> Currently the user gets a TimerService to register timers that will in the
> future call the onTimer method. It is not completely obvious to me how
> would I implement a function that needs to trigger two types of callbacks
> in the future. If I get only one onTimer method I should be able to pass in
> some sort of parameter or flag so I can branch in my onTimer
> implementation.
>
> As parameters are not supported I am left with states that are scoped to
> the keys which is also pretty useless if I want to trigger different timed
> actions for the same keys.
>
> I know this is quite tricky but I see some alternative options:
>  - The register timer method returns a unique (per key) timer id, so we can
> associate state with this id to fetch info about the timer registered. (We
> could also remove timers with this id and maybe add methods to remove all
> for the current key)
>  - Allow the users to pass a custom parameter when registering the
> callback, and the parameter would be passed to the onTimer method
>  - Allow users to pass custom callback functions when registering the
> timers, but this would mean we have to support some sort of context for
> accessing the state (like the window context before)
>  - We could go for an annotation based API like in beam but thats probably
> not good to mix in the current ones
>
> I personally prefer the first one.
>
> What do you think?
>
> Regards,
> Gyula
>

[Jamie Grier]

Hi guys,

Good points, Gyula.  I think it would be much easier on a user if there
could be multiple timers in flight per key.  I prefer the second approach,
though, where a user associates some bit of metadata with the timer and we
pass it back to them in the onTimer() callback, otherwise they are forced
to maintain this state themselves.

It looks to me like somehow exposing the namespaces, even if it's simpler
and just a string, is the way to go.

I'm really excited by this guys!  I think the TimelyFlatMap and
TimelyCoFlatMap are going to get a LOT of use.  This is gonna make a lot of
people happy.

-Jamie


On Fri, Oct 28, 2016 at 1:58 PM, Aljoscha Krettek <[hidden email]>
wrote:

> Hi Gyula,
> if you look at the internal API you'll notice that it is pretty much like
> your second proposal. Just for reference, the interface is roughly this:
>
> public interface InternalTimerService<N> {
>   long currentProcessingTime();
>   long currentWatermark();
>   void registerProcessingTimeTimer(N namespace, long time);
>   void deleteProcessingTimeTimer(N namespace, long time);
>   void registerEventTimeTimer(N namespace, long time);
>   void deleteEventTimeTimer(N namespace, long time);
> }
>
> that namespace bit can be anything for which you can provide a
> TypeSerializer.
>
> IMHO, this goes back a bit to the discussion about adding a low level
> operator-like interface that allows pretty much anything a custom operator
> can do but with a defined, stable interface. The internal operator
> interface is somewhat in flux, still, so I wouldn't recommend people to use
> it directly.
>
> The only thing missing, really, from TimelyFlatMap is access to namespaces
> for state and timers. With that, you could implement even the
> WindowOperator as a TimelyFlatMap since I worked on abstracting everything
> that it uses away behind interfaces that any operator can use. The last
> pice, the generic timer API went in last, of course. :-)
>
> Cheers,
> Aljoscha
>
> On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <[hidden email]> wrote:
>
> > Hello,
> >
> > I was thinking about the methods provided by the timely functions and the
> > timerservice and I am wondering if it makes sense to change them a little
> > so they can cover a wider set of use case. Maybe I missed something
> > completely obvious so please shoot me down in that case :)
> >
> > Currently the user gets a TimerService to register timers that will in
> the
> > future call the onTimer method. It is not completely obvious to me how
> > would I implement a function that needs to trigger two types of callbacks
> > in the future. If I get only one onTimer method I should be able to pass
> in
> > some sort of parameter or flag so I can branch in my onTimer
> > implementation.
> >
> > As parameters are not supported I am left with states that are scoped to
> > the keys which is also pretty useless if I want to trigger different
> timed
> > actions for the same keys.
> >
> > I know this is quite tricky but I see some alternative options:
> >  - The register timer method returns a unique (per key) timer id, so we
> can
> > associate state with this id to fetch info about the timer registered.
> (We
> > could also remove timers with this id and maybe add methods to remove all
> > for the current key)
> >  - Allow the users to pass a custom parameter when registering the
> > callback, and the parameter would be passed to the onTimer method
> >  - Allow users to pass custom callback functions when registering the
> > timers, but this would mean we have to support some sort of context for
> > accessing the state (like the window context before)
> >  - We could go for an annotation based API like in beam but thats
> probably
> > not good to mix in the current ones
> >
> > I personally prefer the first one.
> >
> > What do you think?
> >
> > Regards,
> > Gyula
> >
>

--

Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier <https://twitter.com/jamiegrier>
[hidden email]

[Gyula Fóra]

Thanks for the feedback guys,

I think exposing the namespace in a simplified form in the user facing API
is I think a very good idea, that already let's the users implement
practically anything they want. Maybe doing it as a simple string as Jamie
suggested would be a nice way to do it and that would serve as a label or
timer-id for the user. Thinking of it as a label/id is probably a much
simpler concept than the "namespace".

Should we open a JIRA for this? Judging from the internal timer service
this should be a fairly straight forward extension as Aljoscha pointed out.

Gyula

Jamie Grier <[hidden email]> ezt írta (időpont: 2016. okt. 29.,
Szo, 15:37):

> Hi guys,
>
> Good points, Gyula.  I think it would be much easier on a user if there
> could be multiple timers in flight per key.  I prefer the second approach,
> though, where a user associates some bit of metadata with the timer and we
> pass it back to them in the onTimer() callback, otherwise they are forced
> to maintain this state themselves.
>
> It looks to me like somehow exposing the namespaces, even if it's simpler
> and just a string, is the way to go.
>
> I'm really excited by this guys!  I think the TimelyFlatMap and
> TimelyCoFlatMap are going to get a LOT of use.  This is gonna make a lot of
> people happy.
>
> -Jamie
>
>
> On Fri, Oct 28, 2016 at 1:58 PM, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > Hi Gyula,
> > if you look at the internal API you'll notice that it is pretty much like
> > your second proposal. Just for reference, the interface is roughly this:
> >
> > public interface InternalTimerService<N> {
> >   long currentProcessingTime();
> >   long currentWatermark();
> >   void registerProcessingTimeTimer(N namespace, long time);
> >   void deleteProcessingTimeTimer(N namespace, long time);
> >   void registerEventTimeTimer(N namespace, long time);
> >   void deleteEventTimeTimer(N namespace, long time);
> > }
> >
> > that namespace bit can be anything for which you can provide a
> > TypeSerializer.
> >
> > IMHO, this goes back a bit to the discussion about adding a low level
> > operator-like interface that allows pretty much anything a custom
> operator
> > can do but with a defined, stable interface. The internal operator
> > interface is somewhat in flux, still, so I wouldn't recommend people to
> use
> > it directly.
> >
> > The only thing missing, really, from TimelyFlatMap is access to
> namespaces
> > for state and timers. With that, you could implement even the
> > WindowOperator as a TimelyFlatMap since I worked on abstracting
> everything
> > that it uses away behind interfaces that any operator can use. The last
> > pice, the generic timer API went in last, of course. :-)
> >
> > Cheers,
> > Aljoscha
> >
> > On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <[hidden email]> wrote:
> >
> > > Hello,
> > >
> > > I was thinking about the methods provided by the timely functions and
> the
> > > timerservice and I am wondering if it makes sense to change them a
> little
> > > so they can cover a wider set of use case. Maybe I missed something
> > > completely obvious so please shoot me down in that case :)
> > >
> > > Currently the user gets a TimerService to register timers that will in
> > the
> > > future call the onTimer method. It is not completely obvious to me how
> > > would I implement a function that needs to trigger two types of
> callbacks
> > > in the future. If I get only one onTimer method I should be able to
> pass
> > in
> > > some sort of parameter or flag so I can branch in my onTimer
> > > implementation.
> > >
> > > As parameters are not supported I am left with states that are scoped
> to
> > > the keys which is also pretty useless if I want to trigger different
> > timed
> > > actions for the same keys.
> > >
> > > I know this is quite tricky but I see some alternative options:
> > >  - The register timer method returns a unique (per key) timer id, so we
> > can
> > > associate state with this id to fetch info about the timer registered.
> > (We
> > > could also remove timers with this id and maybe add methods to remove
> all
> > > for the current key)
> > >  - Allow the users to pass a custom parameter when registering the
> > > callback, and the parameter would be passed to the onTimer method
> > >  - Allow users to pass custom callback functions when registering the
> > > timers, but this would mean we have to support some sort of context for
> > > accessing the state (like the window context before)
> > >  - We could go for an annotation based API like in beam but thats
> > probably
> > > not good to mix in the current ones
> > >
> > > I personally prefer the first one.
> > >
> > > What do you think?
> > >
> > > Regards,
> > > Gyula
> > >
> >
>
>
>
> --
>
> Jamie Grier
> data Artisans, Director of Applications Engineering
> @jamiegrier <https://twitter.com/jamiegrier>
> [hidden email]
>

[Aljoscha Krettek-2]

Hi,
yes, I think exposing a simple form of namespaces as String would be good.
By the way, do we also need access to state with namespaces then?

@Gyula: Please go ahead and open the Jira issue.

Cheers,
Aljoscha

On Sat, 29 Oct 2016 at 17:28 Gyula Fóra <[hidden email]> wrote:

> Thanks for the feedback guys,
>
> I think exposing the namespace in a simplified form in the user facing API
> is I think a very good idea, that already let's the users implement
> practically anything they want. Maybe doing it as a simple string as Jamie
> suggested would be a nice way to do it and that would serve as a label or
> timer-id for the user. Thinking of it as a label/id is probably a much
> simpler concept than the "namespace".
>
> Should we open a JIRA for this? Judging from the internal timer service
> this should be a fairly straight forward extension as Aljoscha pointed out.
>
> Gyula
>
> Jamie Grier <[hidden email]> ezt írta (időpont: 2016. okt. 29.,
> Szo, 15:37):
>
> > Hi guys,
> >
> > Good points, Gyula.  I think it would be much easier on a user if there
> > could be multiple timers in flight per key.  I prefer the second
> approach,
> > though, where a user associates some bit of metadata with the timer and
> we
> > pass it back to them in the onTimer() callback, otherwise they are forced
> > to maintain this state themselves.
> >
> > It looks to me like somehow exposing the namespaces, even if it's simpler
> > and just a string, is the way to go.
> >
> > I'm really excited by this guys!  I think the TimelyFlatMap and
> > TimelyCoFlatMap are going to get a LOT of use.  This is gonna make a lot
> of
> > people happy.
> >
> > -Jamie
> >
> >
> > On Fri, Oct 28, 2016 at 1:58 PM, Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> > > Hi Gyula,
> > > if you look at the internal API you'll notice that it is pretty much
> like
> > > your second proposal. Just for reference, the interface is roughly
> this:
> > >
> > > public interface InternalTimerService<N> {
> > >   long currentProcessingTime();
> > >   long currentWatermark();
> > >   void registerProcessingTimeTimer(N namespace, long time);
> > >   void deleteProcessingTimeTimer(N namespace, long time);
> > >   void registerEventTimeTimer(N namespace, long time);
> > >   void deleteEventTimeTimer(N namespace, long time);
> > > }
> > >
> > > that namespace bit can be anything for which you can provide a
> > > TypeSerializer.
> > >
> > > IMHO, this goes back a bit to the discussion about adding a low level
> > > operator-like interface that allows pretty much anything a custom
> > operator
> > > can do but with a defined, stable interface. The internal operator
> > > interface is somewhat in flux, still, so I wouldn't recommend people to
> > use
> > > it directly.
> > >
> > > The only thing missing, really, from TimelyFlatMap is access to
> > namespaces
> > > for state and timers. With that, you could implement even the
> > > WindowOperator as a TimelyFlatMap since I worked on abstracting
> > everything
> > > that it uses away behind interfaces that any operator can use. The last
> > > pice, the generic timer API went in last, of course. :-)
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <[hidden email]> wrote:
> > >
> > > > Hello,
> > > >
> > > > I was thinking about the methods provided by the timely functions and
> > the
> > > > timerservice and I am wondering if it makes sense to change them a
> > little
> > > > so they can cover a wider set of use case. Maybe I missed something
> > > > completely obvious so please shoot me down in that case :)
> > > >
> > > > Currently the user gets a TimerService to register timers that will
> in
> > > the
> > > > future call the onTimer method. It is not completely obvious to me
> how
> > > > would I implement a function that needs to trigger two types of
> > callbacks
> > > > in the future. If I get only one onTimer method I should be able to
> > pass
> > > in
> > > > some sort of parameter or flag so I can branch in my onTimer
> > > > implementation.
> > > >
> > > > As parameters are not supported I am left with states that are scoped
> > to
> > > > the keys which is also pretty useless if I want to trigger different
> > > timed
> > > > actions for the same keys.
> > > >
> > > > I know this is quite tricky but I see some alternative options:
> > > >  - The register timer method returns a unique (per key) timer id, so
> we
> > > can
> > > > associate state with this id to fetch info about the timer
> registered.
> > > (We
> > > > could also remove timers with this id and maybe add methods to remove
> > all
> > > > for the current key)
> > > >  - Allow the users to pass a custom parameter when registering the
> > > > callback, and the parameter would be passed to the onTimer method
> > > >  - Allow users to pass custom callback functions when registering the
> > > > timers, but this would mean we have to support some sort of context
> for
> > > > accessing the state (like the window context before)
> > > >  - We could go for an annotation based API like in beam but thats
> > > probably
> > > > not good to mix in the current ones
> > > >
> > > > I personally prefer the first one.
> > > >
> > > > What do you think?
> > > >
> > > > Regards,
> > > > Gyula
> > > >
> > >
> >
> >
> >
> > --
> >
> > Jamie Grier
> > data Artisans, Director of Applications Engineering
> > @jamiegrier <https://twitter.com/jamiegrier>
> > [hidden email]
> >
>

[Gyula Fóra]

Hi,
I opened this:
https://issues.apache.org/jira/browse/FLINK-4992

With this users can implement practically anything depending how they use
the parameter so changing the state access logic is not necessary.

Cheers,
Gyula

Aljoscha Krettek <[hidden email]> ezt írta (időpont: 2016. nov. 1., K,
16:39):

> Hi,
> yes, I think exposing a simple form of namespaces as String would be good.
> By the way, do we also need access to state with namespaces then?
>
> @Gyula: Please go ahead and open the Jira issue.
>
> Cheers,
> Aljoscha
>
> On Sat, 29 Oct 2016 at 17:28 Gyula Fóra <[hidden email]> wrote:
>
> > Thanks for the feedback guys,
> >
> > I think exposing the namespace in a simplified form in the user facing
> API
> > is I think a very good idea, that already let's the users implement
> > practically anything they want. Maybe doing it as a simple string as
> Jamie
> > suggested would be a nice way to do it and that would serve as a label or
> > timer-id for the user. Thinking of it as a label/id is probably a much
> > simpler concept than the "namespace".
> >
> > Should we open a JIRA for this? Judging from the internal timer service
> > this should be a fairly straight forward extension as Aljoscha pointed
> out.
> >
> > Gyula
> >
> > Jamie Grier <[hidden email]> ezt írta (időpont: 2016. okt. 29.,
> > Szo, 15:37):
> >
> > > Hi guys,
> > >
> > > Good points, Gyula.  I think it would be much easier on a user if there
> > > could be multiple timers in flight per key.  I prefer the second
> > approach,
> > > though, where a user associates some bit of metadata with the timer and
> > we
> > > pass it back to them in the onTimer() callback, otherwise they are
> forced
> > > to maintain this state themselves.
> > >
> > > It looks to me like somehow exposing the namespaces, even if it's
> simpler
> > > and just a string, is the way to go.
> > >
> > > I'm really excited by this guys!  I think the TimelyFlatMap and
> > > TimelyCoFlatMap are going to get a LOT of use.  This is gonna make a
> lot
> > of
> > > people happy.
> > >
> > > -Jamie
> > >
> > >
> > > On Fri, Oct 28, 2016 at 1:58 PM, Aljoscha Krettek <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi Gyula,
> > > > if you look at the internal API you'll notice that it is pretty much
> > like
> > > > your second proposal. Just for reference, the interface is roughly
> > this:
> > > >
> > > > public interface InternalTimerService<N> {
> > > >   long currentProcessingTime();
> > > >   long currentWatermark();
> > > >   void registerProcessingTimeTimer(N namespace, long time);
> > > >   void deleteProcessingTimeTimer(N namespace, long time);
> > > >   void registerEventTimeTimer(N namespace, long time);
> > > >   void deleteEventTimeTimer(N namespace, long time);
> > > > }
> > > >
> > > > that namespace bit can be anything for which you can provide a
> > > > TypeSerializer.
> > > >
> > > > IMHO, this goes back a bit to the discussion about adding a low level
> > > > operator-like interface that allows pretty much anything a custom
> > > operator
> > > > can do but with a defined, stable interface. The internal operator
> > > > interface is somewhat in flux, still, so I wouldn't recommend people
> to
> > > use
> > > > it directly.
> > > >
> > > > The only thing missing, really, from TimelyFlatMap is access to
> > > namespaces
> > > > for state and timers. With that, you could implement even the
> > > > WindowOperator as a TimelyFlatMap since I worked on abstracting
> > > everything
> > > > that it uses away behind interfaces that any operator can use. The
> last
> > > > pice, the generic timer API went in last, of course. :-)
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Fri, 28 Oct 2016 at 16:55 Gyula Fóra <[hidden email]> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I was thinking about the methods provided by the timely functions
> and
> > > the
> > > > > timerservice and I am wondering if it makes sense to change them a
> > > little
> > > > > so they can cover a wider set of use case. Maybe I missed something
> > > > > completely obvious so please shoot me down in that case :)
> > > > >
> > > > > Currently the user gets a TimerService to register timers that will
> > in
> > > > the
> > > > > future call the onTimer method. It is not completely obvious to me
> > how
> > > > > would I implement a function that needs to trigger two types of
> > > callbacks
> > > > > in the future. If I get only one onTimer method I should be able to
> > > pass
> > > > in
> > > > > some sort of parameter or flag so I can branch in my onTimer
> > > > > implementation.
> > > > >
> > > > > As parameters are not supported I am left with states that are
> scoped
> > > to
> > > > > the keys which is also pretty useless if I want to trigger
> different
> > > > timed
> > > > > actions for the same keys.
> > > > >
> > > > > I know this is quite tricky but I see some alternative options:
> > > > >  - The register timer method returns a unique (per key) timer id,
> so
> > we
> > > > can
> > > > > associate state with this id to fetch info about the timer
> > registered.
> > > > (We
> > > > > could also remove timers with this id and maybe add methods to
> remove
> > > all
> > > > > for the current key)
> > > > >  - Allow the users to pass a custom parameter when registering the
> > > > > callback, and the parameter would be passed to the onTimer method
> > > > >  - Allow users to pass custom callback functions when registering
> the
> > > > > timers, but this would mean we have to support some sort of context
> > for
> > > > > accessing the state (like the window context before)
> > > > >  - We could go for an annotation based API like in beam but thats
> > > > probably
> > > > > not good to mix in the current ones
> > > > >
> > > > > I personally prefer the first one.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Regards,
> > > > > Gyula
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Jamie Grier
> > > data Artisans, Director of Applications Engineering
> > > @jamiegrier <https://twitter.com/jamiegrier>
> > > [hidden email]
> > >
> >
>