Question about SlidingPreReducers

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about SlidingPreReducers

Aljoscha Krettek-2
Hi,
one of my notorious (by now) investigative questions: How do these
work? To meet it seems that they do not perform any actual pre
aggregation but just keep all the elements in a LinkedList
(non-grouped case) or HashMap (grouped-case) and aggregate the
elements when the window is to be emitted.

Am I missing something again? :D

Cheers,
Aljoscha
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Gyula Fóra-2
Hey,

They actually work :P Although I have to admit I need to do some
refactoring of the method names and parameters.

I made some quick refactoring and added some comments for the key methods:

https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96

https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105

https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132

Those three parts contain the key logic, and now the method names should
make more sense as well :)

Cheers,
Gyula





On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <[hidden email]>
wrote:

> Hi,
> one of my notorious (by now) investigative questions: How do these
> work? To meet it seems that they do not perform any actual pre
> aggregation but just keep all the elements in a LinkedList
> (non-grouped case) or HashMap (grouped-case) and aggregate the
> elements when the window is to be emitted.
>
> Am I missing something again? :D
>
> Cheers,
> Aljoscha
>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Aljoscha Krettek-2
Oh, I saw them, but in updateCurrent() (both grouped and non-grouped)
the else path is never taken when I run the windowed WordCount
example.

I have a branch where I just throw a RuntimeException in the else
branches and the whole things still works: mvn verify doesn't fail,
the examples still run. When looking at the behaviour of the
functions, it seems clear to me that the else path can never be taken.

On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]> wrote:

> Hey,
>
> They actually work :P Although I have to admit I need to do some
> refactoring of the method names and parameters.
>
> I made some quick refactoring and added some comments for the key methods:
>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
>
> Those three parts contain the key logic, and now the method names should
> make more sense as well :)
>
> Cheers,
> Gyula
>
>
>
>
>
> On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <[hidden email]>
> wrote:
>
>> Hi,
>> one of my notorious (by now) investigative questions: How do these
>> work? To meet it seems that they do not perform any actual pre
>> aggregation but just keep all the elements in a LinkedList
>> (non-grouped case) or HashMap (grouped-case) and aggregate the
>> elements when the window is to be emitted.
>>
>> Am I missing something again? :D
>>
>> Cheers,
>> Aljoscha
>>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Gyula Fóra-2
Hey,
You are right, so there seems to be some error in the logic that checks for
the end of the slide.
I checked the count pre-reducer and the error there is clear, but if I fix
it it breaks some tests so I think some other stuff must not work properly
either.

I cannot work on this until I get back from holiday, it would be good if
you could look into it.

Cheers,
Gyula

On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <[hidden email]>
wrote:

> Oh, I saw them, but in updateCurrent() (both grouped and non-grouped)
> the else path is never taken when I run the windowed WordCount
> example.
>
> I have a branch where I just throw a RuntimeException in the else
> branches and the whole things still works: mvn verify doesn't fail,
> the examples still run. When looking at the behaviour of the
> functions, it seems clear to me that the else path can never be taken.
>
> On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]> wrote:
> > Hey,
> >
> > They actually work :P Although I have to admit I need to do some
> > refactoring of the method names and parameters.
> >
> > I made some quick refactoring and added some comments for the key
> methods:
> >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
> >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
> >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
> >
> > Those three parts contain the key logic, and now the method names should
> > make more sense as well :)
> >
> > Cheers,
> > Gyula
> >
> >
> >
> >
> >
> > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> >> Hi,
> >> one of my notorious (by now) investigative questions: How do these
> >> work? To meet it seems that they do not perform any actual pre
> >> aggregation but just keep all the elements in a LinkedList
> >> (non-grouped case) or HashMap (grouped-case) and aggregate the
> >> elements when the window is to be emitted.
> >>
> >> Am I missing something again? :D
> >>
> >> Cheers,
> >> Aljoscha
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Aljoscha Krettek-2
I am on it, yes. :D
On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email]> wrote:

> Hey,
> You are right, so there seems to be some error in the logic that checks for
> the end of the slide.
> I checked the count pre-reducer and the error there is clear, but if I fix
> it it breaks some tests so I think some other stuff must not work properly
> either.
>
> I cannot work on this until I get back from holiday, it would be good if
> you could look into it.
>
> Cheers,
> Gyula
>
> On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > Oh, I saw them, but in updateCurrent() (both grouped and non-grouped)
> > the else path is never taken when I run the windowed WordCount
> > example.
> >
> > I have a branch where I just throw a RuntimeException in the else
> > branches and the whole things still works: mvn verify doesn't fail,
> > the examples still run. When looking at the behaviour of the
> > functions, it seems clear to me that the else path can never be taken.
> >
> > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]> wrote:
> > > Hey,
> > >
> > > They actually work :P Although I have to admit I need to do some
> > > refactoring of the method names and parameters.
> > >
> > > I made some quick refactoring and added some comments for the key
> > methods:
> > >
> > >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
> > >
> > >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
> > >
> > >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
> > >
> > > Those three parts contain the key logic, and now the method names
> should
> > > make more sense as well :)
> > >
> > > Cheers,
> > > Gyula
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <[hidden email]
> >
> > > wrote:
> > >
> > >> Hi,
> > >> one of my notorious (by now) investigative questions: How do these
> > >> work? To meet it seems that they do not perform any actual pre
> > >> aggregation but just keep all the elements in a LinkedList
> > >> (non-grouped case) or HashMap (grouped-case) and aggregate the
> > >> elements when the window is to be emitted.
> > >>
> > >> Am I missing something again? :D
> > >>
> > >> Cheers,
> > >> Aljoscha
> > >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Szabó Péter
Hey,

our intern, Pablo pointed out that there is some problem with mixed
windowing policies. When you write
  ...
  .window(Count ...)
  .every(Time ...)
  .mapWindow(...)
  ...

The result makes no sense, as the window is not of the specified length.
Maybe, there is some conflict between Time and Count logic. Is this a
related problem to yours?

Peter

2015-04-29 21:15 GMT+02:00 Aljoscha Krettek <[hidden email]>:

> I am on it, yes. :D
> On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email]> wrote:
>
> > Hey,
> > You are right, so there seems to be some error in the logic that checks
> for
> > the end of the slide.
> > I checked the count pre-reducer and the error there is clear, but if I
> fix
> > it it breaks some tests so I think some other stuff must not work
> properly
> > either.
> >
> > I cannot work on this until I get back from holiday, it would be good if
> > you could look into it.
> >
> > Cheers,
> > Gyula
> >
> > On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> > > Oh, I saw them, but in updateCurrent() (both grouped and non-grouped)
> > > the else path is never taken when I run the windowed WordCount
> > > example.
> > >
> > > I have a branch where I just throw a RuntimeException in the else
> > > branches and the whole things still works: mvn verify doesn't fail,
> > > the examples still run. When looking at the behaviour of the
> > > functions, it seems clear to me that the else path can never be taken.
> > >
> > > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]> wrote:
> > > > Hey,
> > > >
> > > > They actually work :P Although I have to admit I need to do some
> > > > refactoring of the method names and parameters.
> > > >
> > > > I made some quick refactoring and added some comments for the key
> > > methods:
> > > >
> > > >
> > >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
> > > >
> > > >
> > >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
> > > >
> > > >
> > >
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
> > > >
> > > > Those three parts contain the key logic, and now the method names
> > should
> > > > make more sense as well :)
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >> one of my notorious (by now) investigative questions: How do these
> > > >> work? To meet it seems that they do not perform any actual pre
> > > >> aggregation but just keep all the elements in a LinkedList
> > > >> (non-grouped case) or HashMap (grouped-case) and aggregate the
> > > >> elements when the window is to be emitted.
> > > >>
> > > >> Am I missing something again? :D
> > > >>
> > > >> Cheers,
> > > >> Aljoscha
> > > >>
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Aljoscha Krettek-2
Hi,
no, I think the two are unrelated. But that's another problem we need
to tackle then.

Cheers,
Aljoscha

On Thu, Apr 30, 2015 at 9:15 AM, Szabó Péter <[hidden email]> wrote:

> Hey,
>
> our intern, Pablo pointed out that there is some problem with mixed
> windowing policies. When you write
>   ...
>   .window(Count ...)
>   .every(Time ...)
>   .mapWindow(...)
>   ...
>
> The result makes no sense, as the window is not of the specified length.
> Maybe, there is some conflict between Time and Count logic. Is this a
> related problem to yours?
>
> Peter
>
> 2015-04-29 21:15 GMT+02:00 Aljoscha Krettek <[hidden email]>:
>
>> I am on it, yes. :D
>> On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email]> wrote:
>>
>> > Hey,
>> > You are right, so there seems to be some error in the logic that checks
>> for
>> > the end of the slide.
>> > I checked the count pre-reducer and the error there is clear, but if I
>> fix
>> > it it breaks some tests so I think some other stuff must not work
>> properly
>> > either.
>> >
>> > I cannot work on this until I get back from holiday, it would be good if
>> > you could look into it.
>> >
>> > Cheers,
>> > Gyula
>> >
>> > On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <[hidden email]>
>> > wrote:
>> >
>> > > Oh, I saw them, but in updateCurrent() (both grouped and non-grouped)
>> > > the else path is never taken when I run the windowed WordCount
>> > > example.
>> > >
>> > > I have a branch where I just throw a RuntimeException in the else
>> > > branches and the whole things still works: mvn verify doesn't fail,
>> > > the examples still run. When looking at the behaviour of the
>> > > functions, it seems clear to me that the else path can never be taken.
>> > >
>> > > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]> wrote:
>> > > > Hey,
>> > > >
>> > > > They actually work :P Although I have to admit I need to do some
>> > > > refactoring of the method names and parameters.
>> > > >
>> > > > I made some quick refactoring and added some comments for the key
>> > > methods:
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
>> > > >
>> > > > Those three parts contain the key logic, and now the method names
>> > should
>> > > > make more sense as well :)
>> > > >
>> > > > Cheers,
>> > > > Gyula
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <
>> [hidden email]
>> > >
>> > > > wrote:
>> > > >
>> > > >> Hi,
>> > > >> one of my notorious (by now) investigative questions: How do these
>> > > >> work? To meet it seems that they do not perform any actual pre
>> > > >> aggregation but just keep all the elements in a LinkedList
>> > > >> (non-grouped case) or HashMap (grouped-case) and aggregate the
>> > > >> elements when the window is to be emitted.
>> > > >>
>> > > >> Am I missing something again? :D
>> > > >>
>> > > >> Cheers,
>> > > >> Aljoscha
>> > > >>
>> > >
>> >
>>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Gyula Fóra
The problem is in the WindowUtils.isParallel policy method. It makes count
policies automatically parallel as well.

On Thursday, April 30, 2015, Aljoscha Krettek <[hidden email]> wrote:

> Hi,
> no, I think the two are unrelated. But that's another problem we need
> to tackle then.
>
> Cheers,
> Aljoscha
>
> On Thu, Apr 30, 2015 at 9:15 AM, Szabó Péter <[hidden email]
> <javascript:;>> wrote:
> > Hey,
> >
> > our intern, Pablo pointed out that there is some problem with mixed
> > windowing policies. When you write
> >   ...
> >   .window(Count ...)
> >   .every(Time ...)
> >   .mapWindow(...)
> >   ...
> >
> > The result makes no sense, as the window is not of the specified length.
> > Maybe, there is some conflict between Time and Count logic. Is this a
> > related problem to yours?
> >
> > Peter
> >
> > 2015-04-29 21:15 GMT+02:00 Aljoscha Krettek <[hidden email]
> <javascript:;>>:
> >
> >> I am on it, yes. :D
> >> On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email] <javascript:;>>
> wrote:
> >>
> >> > Hey,
> >> > You are right, so there seems to be some error in the logic that
> checks
> >> for
> >> > the end of the slide.
> >> > I checked the count pre-reducer and the error there is clear, but if I
> >> fix
> >> > it it breaks some tests so I think some other stuff must not work
> >> properly
> >> > either.
> >> >
> >> > I cannot work on this until I get back from holiday, it would be good
> if
> >> > you could look into it.
> >> >
> >> > Cheers,
> >> > Gyula
> >> >
> >> > On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <
> [hidden email] <javascript:;>>
> >> > wrote:
> >> >
> >> > > Oh, I saw them, but in updateCurrent() (both grouped and
> non-grouped)
> >> > > the else path is never taken when I run the windowed WordCount
> >> > > example.
> >> > >
> >> > > I have a branch where I just throw a RuntimeException in the else
> >> > > branches and the whole things still works: mvn verify doesn't fail,
> >> > > the examples still run. When looking at the behaviour of the
> >> > > functions, it seems clear to me that the else path can never be
> taken.
> >> > >
> >> > > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]
> <javascript:;>> wrote:
> >> > > > Hey,
> >> > > >
> >> > > > They actually work :P Although I have to admit I need to do some
> >> > > > refactoring of the method names and parameters.
> >> > > >
> >> > > > I made some quick refactoring and added some comments for the key
> >> > > methods:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
> >> > > >
> >> > > > Those three parts contain the key logic, and now the method names
> >> > should
> >> > > > make more sense as well :)
> >> > > >
> >> > > > Cheers,
> >> > > > Gyula
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <
> >> [hidden email] <javascript:;>
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > >> Hi,
> >> > > >> one of my notorious (by now) investigative questions: How do
> these
> >> > > >> work? To meet it seems that they do not perform any actual pre
> >> > > >> aggregation but just keep all the elements in a LinkedList
> >> > > >> (non-grouped case) or HashMap (grouped-case) and aggregate the
> >> > > >> elements when the window is to be emitted.
> >> > > >>
> >> > > >> Am I missing something again? :D
> >> > > >>
> >> > > >> Cheers,
> >> > > >> Aljoscha
> >> > > >>
> >> > >
> >> >
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Aljoscha Krettek-2
@Gyula: Are you referring to the pre-aggregator or the thing Peter mentioned?

On Thu, Apr 30, 2015 at 11:12 AM, Gyula Fóra <[hidden email]> wrote:

> The problem is in the WindowUtils.isParallel policy method. It makes count
> policies automatically parallel as well.
>
> On Thursday, April 30, 2015, Aljoscha Krettek <[hidden email]> wrote:
>
>> Hi,
>> no, I think the two are unrelated. But that's another problem we need
>> to tackle then.
>>
>> Cheers,
>> Aljoscha
>>
>> On Thu, Apr 30, 2015 at 9:15 AM, Szabó Péter <[hidden email]
>> <javascript:;>> wrote:
>> > Hey,
>> >
>> > our intern, Pablo pointed out that there is some problem with mixed
>> > windowing policies. When you write
>> >   ...
>> >   .window(Count ...)
>> >   .every(Time ...)
>> >   .mapWindow(...)
>> >   ...
>> >
>> > The result makes no sense, as the window is not of the specified length.
>> > Maybe, there is some conflict between Time and Count logic. Is this a
>> > related problem to yours?
>> >
>> > Peter
>> >
>> > 2015-04-29 21:15 GMT+02:00 Aljoscha Krettek <[hidden email]
>> <javascript:;>>:
>> >
>> >> I am on it, yes. :D
>> >> On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email] <javascript:;>>
>> wrote:
>> >>
>> >> > Hey,
>> >> > You are right, so there seems to be some error in the logic that
>> checks
>> >> for
>> >> > the end of the slide.
>> >> > I checked the count pre-reducer and the error there is clear, but if I
>> >> fix
>> >> > it it breaks some tests so I think some other stuff must not work
>> >> properly
>> >> > either.
>> >> >
>> >> > I cannot work on this until I get back from holiday, it would be good
>> if
>> >> > you could look into it.
>> >> >
>> >> > Cheers,
>> >> > Gyula
>> >> >
>> >> > On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <
>> [hidden email] <javascript:;>>
>> >> > wrote:
>> >> >
>> >> > > Oh, I saw them, but in updateCurrent() (both grouped and
>> non-grouped)
>> >> > > the else path is never taken when I run the windowed WordCount
>> >> > > example.
>> >> > >
>> >> > > I have a branch where I just throw a RuntimeException in the else
>> >> > > branches and the whole things still works: mvn verify doesn't fail,
>> >> > > the examples still run. When looking at the behaviour of the
>> >> > > functions, it seems clear to me that the else path can never be
>> taken.
>> >> > >
>> >> > > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]
>> <javascript:;>> wrote:
>> >> > > > Hey,
>> >> > > >
>> >> > > > They actually work :P Although I have to admit I need to do some
>> >> > > > refactoring of the method names and parameters.
>> >> > > >
>> >> > > > I made some quick refactoring and added some comments for the key
>> >> > > methods:
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
>> >> > > >
>> >> > > > Those three parts contain the key logic, and now the method names
>> >> > should
>> >> > > > make more sense as well :)
>> >> > > >
>> >> > > > Cheers,
>> >> > > > Gyula
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <
>> >> [hidden email] <javascript:;>
>> >> > >
>> >> > > > wrote:
>> >> > > >
>> >> > > >> Hi,
>> >> > > >> one of my notorious (by now) investigative questions: How do
>> these
>> >> > > >> work? To meet it seems that they do not perform any actual pre
>> >> > > >> aggregation but just keep all the elements in a LinkedList
>> >> > > >> (non-grouped case) or HashMap (grouped-case) and aggregate the
>> >> > > >> elements when the window is to be emitted.
>> >> > > >>
>> >> > > >> Am I missing something again? :D
>> >> > > >>
>> >> > > >> Cheers,
>> >> > > >> Aljoscha
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Gyula Fóra
I'm referring to Peter's problem. If we just create more count discretizers
it doesn't really break the semantics given the network guarantees but it
is not very intuitive.

On Thursday, April 30, 2015, Aljoscha Krettek <[hidden email]> wrote:

> @Gyula: Are you referring to the pre-aggregator or the thing Peter
> mentioned?
>
> On Thu, Apr 30, 2015 at 11:12 AM, Gyula Fóra <[hidden email]
> <javascript:;>> wrote:
> > The problem is in the WindowUtils.isParallel policy method. It makes
> count
> > policies automatically parallel as well.
> >
> > On Thursday, April 30, 2015, Aljoscha Krettek <[hidden email]
> <javascript:;>> wrote:
> >
> >> Hi,
> >> no, I think the two are unrelated. But that's another problem we need
> >> to tackle then.
> >>
> >> Cheers,
> >> Aljoscha
> >>
> >> On Thu, Apr 30, 2015 at 9:15 AM, Szabó Péter <[hidden email]
> <javascript:;>
> >> <javascript:;>> wrote:
> >> > Hey,
> >> >
> >> > our intern, Pablo pointed out that there is some problem with mixed
> >> > windowing policies. When you write
> >> >   ...
> >> >   .window(Count ...)
> >> >   .every(Time ...)
> >> >   .mapWindow(...)
> >> >   ...
> >> >
> >> > The result makes no sense, as the window is not of the specified
> length.
> >> > Maybe, there is some conflict between Time and Count logic. Is this a
> >> > related problem to yours?
> >> >
> >> > Peter
> >> >
> >> > 2015-04-29 21:15 GMT+02:00 Aljoscha Krettek <[hidden email]
> <javascript:;>
> >> <javascript:;>>:
> >> >
> >> >> I am on it, yes. :D
> >> >> On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email]
> <javascript:;> <javascript:;>>
> >> wrote:
> >> >>
> >> >> > Hey,
> >> >> > You are right, so there seems to be some error in the logic that
> >> checks
> >> >> for
> >> >> > the end of the slide.
> >> >> > I checked the count pre-reducer and the error there is clear, but
> if I
> >> >> fix
> >> >> > it it breaks some tests so I think some other stuff must not work
> >> >> properly
> >> >> > either.
> >> >> >
> >> >> > I cannot work on this until I get back from holiday, it would be
> good
> >> if
> >> >> > you could look into it.
> >> >> >
> >> >> > Cheers,
> >> >> > Gyula
> >> >> >
> >> >> > On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <
> >> [hidden email] <javascript:;> <javascript:;>>
> >> >> > wrote:
> >> >> >
> >> >> > > Oh, I saw them, but in updateCurrent() (both grouped and
> >> non-grouped)
> >> >> > > the else path is never taken when I run the windowed WordCount
> >> >> > > example.
> >> >> > >
> >> >> > > I have a branch where I just throw a RuntimeException in the else
> >> >> > > branches and the whole things still works: mvn verify doesn't
> fail,
> >> >> > > the examples still run. When looking at the behaviour of the
> >> >> > > functions, it seems clear to me that the else path can never be
> >> taken.
> >> >> > >
> >> >> > > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]
> <javascript:;>
> >> <javascript:;>> wrote:
> >> >> > > > Hey,
> >> >> > > >
> >> >> > > > They actually work :P Although I have to admit I need to do
> some
> >> >> > > > refactoring of the method names and parameters.
> >> >> > > >
> >> >> > > > I made some quick refactoring and added some comments for the
> key
> >> >> > > methods:
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
> >> >> > > >
> >> >> > > > Those three parts contain the key logic, and now the method
> names
> >> >> > should
> >> >> > > > make more sense as well :)
> >> >> > > >
> >> >> > > > Cheers,
> >> >> > > > Gyula
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <
> >> >> [hidden email] <javascript:;> <javascript:;>
> >> >> > >
> >> >> > > > wrote:
> >> >> > > >
> >> >> > > >> Hi,
> >> >> > > >> one of my notorious (by now) investigative questions: How do
> >> these
> >> >> > > >> work? To meet it seems that they do not perform any actual pre
> >> >> > > >> aggregation but just keep all the elements in a LinkedList
> >> >> > > >> (non-grouped case) or HashMap (grouped-case) and aggregate the
> >> >> > > >> elements when the window is to be emitted.
> >> >> > > >>
> >> >> > > >> Am I missing something again? :D
> >> >> > > >>
> >> >> > > >> Cheers,
> >> >> > > >> Aljoscha
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Question about SlidingPreReducers

Szabó Péter
So, would it be a reasonable solution to just modify the
WindowUtils.isParallelPolicy() method to return false in case of "eviction
instanceof CountEvictionPolicy && trigger instanceof TimeTriggerPolicy" ?

2015-04-30 12:21 GMT+02:00 Gyula Fóra <[hidden email]>:

> I'm referring to Peter's problem. If we just create more count discretizers
> it doesn't really break the semantics given the network guarantees but it
> is not very intuitive.
>
> On Thursday, April 30, 2015, Aljoscha Krettek <[hidden email]> wrote:
>
> > @Gyula: Are you referring to the pre-aggregator or the thing Peter
> > mentioned?
> >
> > On Thu, Apr 30, 2015 at 11:12 AM, Gyula Fóra <[hidden email]
> > <javascript:;>> wrote:
> > > The problem is in the WindowUtils.isParallel policy method. It makes
> > count
> > > policies automatically parallel as well.
> > >
> > > On Thursday, April 30, 2015, Aljoscha Krettek <[hidden email]
> > <javascript:;>> wrote:
> > >
> > >> Hi,
> > >> no, I think the two are unrelated. But that's another problem we need
> > >> to tackle then.
> > >>
> > >> Cheers,
> > >> Aljoscha
> > >>
> > >> On Thu, Apr 30, 2015 at 9:15 AM, Szabó Péter <
> [hidden email]
> > <javascript:;>
> > >> <javascript:;>> wrote:
> > >> > Hey,
> > >> >
> > >> > our intern, Pablo pointed out that there is some problem with mixed
> > >> > windowing policies. When you write
> > >> >   ...
> > >> >   .window(Count ...)
> > >> >   .every(Time ...)
> > >> >   .mapWindow(...)
> > >> >   ...
> > >> >
> > >> > The result makes no sense, as the window is not of the specified
> > length.
> > >> > Maybe, there is some conflict between Time and Count logic. Is this
> a
> > >> > related problem to yours?
> > >> >
> > >> > Peter
> > >> >
> > >> > 2015-04-29 21:15 GMT+02:00 Aljoscha Krettek <[hidden email]
> > <javascript:;>
> > >> <javascript:;>>:
> > >> >
> > >> >> I am on it, yes. :D
> > >> >> On Apr 29, 2015 8:31 PM, "Gyula Fóra" <[hidden email]
> > <javascript:;> <javascript:;>>
> > >> wrote:
> > >> >>
> > >> >> > Hey,
> > >> >> > You are right, so there seems to be some error in the logic that
> > >> checks
> > >> >> for
> > >> >> > the end of the slide.
> > >> >> > I checked the count pre-reducer and the error there is clear, but
> > if I
> > >> >> fix
> > >> >> > it it breaks some tests so I think some other stuff must not work
> > >> >> properly
> > >> >> > either.
> > >> >> >
> > >> >> > I cannot work on this until I get back from holiday, it would be
> > good
> > >> if
> > >> >> > you could look into it.
> > >> >> >
> > >> >> > Cheers,
> > >> >> > Gyula
> > >> >> >
> > >> >> > On Wed, Apr 29, 2015 at 7:09 PM, Aljoscha Krettek <
> > >> [hidden email] <javascript:;> <javascript:;>>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Oh, I saw them, but in updateCurrent() (both grouped and
> > >> non-grouped)
> > >> >> > > the else path is never taken when I run the windowed WordCount
> > >> >> > > example.
> > >> >> > >
> > >> >> > > I have a branch where I just throw a RuntimeException in the
> else
> > >> >> > > branches and the whole things still works: mvn verify doesn't
> > fail,
> > >> >> > > the examples still run. When looking at the behaviour of the
> > >> >> > > functions, it seems clear to me that the else path can never be
> > >> taken.
> > >> >> > >
> > >> >> > > On Wed, Apr 29, 2015 at 6:15 PM, Gyula Fóra <[hidden email]
> > <javascript:;>
> > >> <javascript:;>> wrote:
> > >> >> > > > Hey,
> > >> >> > > >
> > >> >> > > > They actually work :P Although I have to admit I need to do
> > some
> > >> >> > > > refactoring of the method names and parameters.
> > >> >> > > >
> > >> >> > > > I made some quick refactoring and added some comments for the
> > key
> > >> >> > > methods:
> > >> >> > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L96
> > >> >> > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L105
> > >> >> > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/windowing/windowbuffer/SlidingPreReducer.java#L132
> > >> >> > > >
> > >> >> > > > Those three parts contain the key logic, and now the method
> > names
> > >> >> > should
> > >> >> > > > make more sense as well :)
> > >> >> > > >
> > >> >> > > > Cheers,
> > >> >> > > > Gyula
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On Wed, Apr 29, 2015 at 4:44 PM, Aljoscha Krettek <
> > >> >> [hidden email] <javascript:;> <javascript:;>
> > >> >> > >
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > >> Hi,
> > >> >> > > >> one of my notorious (by now) investigative questions: How do
> > >> these
> > >> >> > > >> work? To meet it seems that they do not perform any actual
> pre
> > >> >> > > >> aggregation but just keep all the elements in a LinkedList
> > >> >> > > >> (non-grouped case) or HashMap (grouped-case) and aggregate
> the
> > >> >> > > >> elements when the window is to be emitted.
> > >> >> > > >>
> > >> >> > > >> Am I missing something again? :D
> > >> >> > > >>
> > >> >> > > >> Cheers,
> > >> >> > > >> Aljoscha
> > >> >> > > >>
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>