I have a datastream
1,2,3,4,5,6,7.... I applied a sliding countWindow as inputStream.keyBy("num").countWindow(2,1) I expect an output as 1,2 2,3 3,4 But am getting an output as 1 1,2 2,3 3,4 Why does the data slide first and then accumulate the window size |
Hi harish,
I will not argue for the correctness of the results, but just tell you why this happens. The countWindow(2, 1) can be regarded as two separate processes: 1) maintain a window whose size *not exceeds* 2 and 2) trigger window evaluation every single record. Actually, in Flink the two processes execute independently and that's why the first record 1 triggered window accumulation in you example. Hope this helps, Xingcan On Thu, Apr 13, 2017 at 4:43 PM, madhairsilence <[hidden email]> wrote: > I have a datastream > 1,2,3,4,5,6,7.... > > I applied a sliding countWindow as > inputStream.keyBy("num").countWindow(2,1) > > I expect an output as > 1,2 > 2,3 > 3,4 > > But am getting an output as > 1 > 1,2 > 2,3 > 3,4 > > Why does the data slide first and then accumulate the window size > > > > -- > View this message in context: http://apache-flink-mailing- > list-archive.1008284.n3.nabble.com/Sliding-Window- > Weird-behaviour-tp17013.html > Sent from the Apache Flink Mailing List archive. mailing list archive at > Nabble.com. > |
Hi Xingcan
Thanks for the answer. But up to my understanding countWindow(4,2) - Should wait for 4 elements (or window not more than 4 element) and once the window is ready, slide two items Now if I have to stopped asking why questions and worry about my current problem, how do I achieve this expected output. Stream : 1,2,3,4,5,6,7,8... Output: 1,2 2,3 3,4 4,5... |
Hi,
You need to implement your own timer. You do this when you create your window by assigning the timer. In your custom timer you would need to implement the desired logic in the onElement method. You can keep a counter that you increment for each element up to your desired number of elements and FIRE only when this value is reaches your threshold after which you want to trigger You can take a look in existing triggers https://github.com/apache/flink/tree/1875cac03042dad4a4c47b0de8364f02fbe457c6/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers -----Original Message----- From: madhairsilence [mailto:[hidden email]] Sent: Thursday, April 13, 2017 12:25 PM To: [hidden email] Subject: Re: Sliding Window - Weird behaviour Hi Xingcan Thanks for the answer. But up to my understanding countWindow(4,2) - Should wait for 4 elements (or window not more than 4 element) and once the window is ready, slide two items Now if I have to stopped asking why questions and worry about my current problem, how do I achieve this expected output. Stream : 1,2,3,4,5,6,7,8... Output: 1,2 2,3 3,4 4,5... -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Sliding-Window-Weird-behaviour-tp17013p17019.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com. |
Free forum by Nabble | Edit this page |