Dynamic SessionWindow gaps

[Dyana Rose]

I have a use case for non-static Session Window gaps.

For example, given a stream of IoT events, each device type could have a
different gap, and that gap could change while sessions are in flight.

I didn't want to have to run a stream processor for each potential gap
length, not to mention the headache of dealing with changing gaps, so I've
implemented a version of SessionWindows that has one major change; in the
assignWindows method it passes the element to a method to extract the
correct sessionTimeout. (current Flink method for reference:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59
)

Preliminary tests show this working as required and I can't be the only
person with this type of use case for session windows.

Will an issue and PR to add this functionality to the SessionWindow classes
be welcome?

Dyana

[Aljoscha Krettek-2]

Hi,

This sounds good! Could you please open a Jira Issue for this and then open a PR against that Jira issue?

Btw, I would suggest to implement this as a new type of assigner, something like DynamicSessionWindows.

Best,
Aljoscha

> On 29. Dec 2017, at 20:54, Dyana Rose <[hidden email]> wrote:
>
> I have a use case for non-static Session Window gaps.
>
> For example, given a stream of IoT events, each device type could have a
> different gap, and that gap could change while sessions are in flight.
>
> I didn't want to have to run a stream processor for each potential gap
> length, not to mention the headache of dealing with changing gaps, so I've
> implemented a version of SessionWindows that has one major change; in the
> assignWindows method it passes the element to a method to extract the
> correct sessionTimeout. (current Flink method for reference:
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59
> )
>
> Preliminary tests show this working as required and I can't be the only
> person with this type of use case for session windows.
>
> Will an issue and PR to add this functionality to the SessionWindow classes
> be welcome?
>
> Dyana

[jincheng sun]

Hi, Dyana dynamic gap is very useful feature, Agree with the suggestion of
Aljoscha, looking forward the PR.

Best, Jincheng

2018-01-02 21:45 GMT+08:00 Aljoscha Krettek <[hidden email]>:

> Hi,
>
> This sounds good! Could you please open a Jira Issue for this and then
> open a PR against that Jira issue?
>
> Btw, I would suggest to implement this as a new type of assigner,
> something like DynamicSessionWindows.
>
> Best,
> Aljoscha
>
> > On 29. Dec 2017, at 20:54, Dyana Rose <[hidden email]> wrote:
> >
> > I have a use case for non-static Session Window gaps.
> >
> > For example, given a stream of IoT events, each device type could have a
> > different gap, and that gap could change while sessions are in flight.
> >
> > I didn't want to have to run a stream processor for each potential gap
> > length, not to mention the headache of dealing with changing gaps, so
> I've
> > implemented a version of SessionWindows that has one major change; in the
> > assignWindows method it passes the element to a method to extract the
> > correct sessionTimeout. (current Flink method for reference:
> > https://github.com/apache/flink/blob/master/flink-
> streaming-java/src/main/java/org/apache/flink/streaming/
> api/windowing/assigners/EventTimeSessionWindows.java#L59
> > )
> >
> > Preliminary tests show this working as required and I can't be the only
> > person with this type of use case for session windows.
> >
> > Will an issue and PR to add this functionality to the SessionWindow
> classes
> > be welcome?
> >
> > Dyana
>
>

[Dyana Rose]

Sounds good, I'm on holiday at the moment. I'll get the ball rolling with a
raised issue when I get in.

Dyana

On 2018-01-02 13:45, Aljoscha Krettek <[hidden email]> wrote:
> Hi,>
>
> This sounds good! Could you please open a Jira Issue for this and then
open a PR against that Jira issue?>
>
> Btw, I would suggest to implement this as a new type of assigner,
something like DynamicSessionWindows.>
>
> Best,>
> Aljoscha>
>
> > On 29. Dec 2017, at 20:54, Dyana Rose <[hidden email]> wrote:>
> > >
> > I have a use case for non-static Session Window gaps.>
> > >
> > For example, given a stream of IoT events, each device type could have
a>
> > different gap, and that gap could change while sessions are in flight.>
> > >
> > I didn't want to have to run a stream processor for each potential gap>
> > length, not to mention the headache of dealing with changing gaps, so
I've>
> > implemented a version of SessionWindows that has one major change; in
the>
> > assignWindows method it passes the element to a method to extract the>
> > correct sessionTimeout. (current Flink method for reference:>
> >
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/assigners/EventTimeSessionWindows.java#L59>

> > )>
> > >
> > Preliminary tests show this working as required and I can't be the
only>
> > person with this type of use case for session windows.>
> > >
> > Will an issue and PR to add this functionality to the SessionWindow
classes>
> > be welcome?>
> > >
> > Dyana>
>
>

[Dyana Rose]

I've put up some implementation information and a link to the working branch for comment while I get the documentation written up:

https://issues.apache.org/jira/browse/FLINK-8384

Thanks,
Dyana

On 2018-01-02 13:54, jincheng sun <[hidden email]> wrote:

> Hi, Dyana dynamic gap is very useful feature, Agree with the suggestion of
> Aljoscha, looking forward the PR.
>
> Best, Jincheng
>
> 2018-01-02 21:45 GMT+08:00 Aljoscha Krettek <[hidden email]>:
>
> > Hi,
> >
> > This sounds good! Could you please open a Jira Issue for this and then
> > open a PR against that Jira issue?
> >
> > Btw, I would suggest to implement this as a new type of assigner,
> > something like DynamicSessionWindows.
> >
> > Best,
> > Aljoscha
> >
> > > On 29. Dec 2017, at 20:54, Dyana Rose <[hidden email]> wrote:
> > >
> > > I have a use case for non-static Session Window gaps.
> > >
> > > For example, given a stream of IoT events, each device type could have a
> > > different gap, and that gap could change while sessions are in flight.
> > >
> > > I didn't want to have to run a stream processor for each potential gap
> > > length, not to mention the headache of dealing with changing gaps, so
> > I've
> > > implemented a version of SessionWindows that has one major change; in the
> > > assignWindows method it passes the element to a method to extract the
> > > correct sessionTimeout. (current Flink method for reference:
> > > https://github.com/apache/flink/blob/master/flink-
> > streaming-java/src/main/java/org/apache/flink/streaming/
> > api/windowing/assigners/EventTimeSessionWindows.java#L59
> > > )
> > >
> > > Preliminary tests show this working as required and I can't be the only
> > > person with this type of use case for session windows.
> > >
> > > Will an issue and PR to add this functionality to the SessionWindow
> > classes
> > > be welcome?
> > >
> > > Dyana
> >
> >
>