Hey!
I started putting together a guide/design document for the streaming operator state interfaces and implementations. The idea would be to create a doc that contains all the details about the implementations so anyone can use it as a reference later. https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing It will probably take me a couple of days to finish it, but in any case feel free to comment. Cheers, Gyula |
Hi,
sorry for the long wait but I finally found the time to read it. It looks good but the later parts of course still need to be fleshed out. I have one comments/questions: In the description of partitioned state you have this sentence: "Operations using partitioned state can also benefit from the partial ordering guarantees that the flink runtime provides, to implement deterministic behaviour." How do we provide the ordering guarantees. I would assume that after a keyBy() the ordering in each partition is arbitrary, unless we add some sorting operator. Cheers, Aljoscha On Thu, 23 Jul 2015 at 15:55 Gyula Fóra <[hidden email]> wrote: > Hey! > I started putting together a guide/design document for the streaming > operator state interfaces and implementations. The idea would be to create > a doc that contains all the details about the implementations so anyone can > use it as a reference later. > > > https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing > > It will probably take me a couple of days to finish it, but in any case > feel free to comment. > > Cheers, > Gyula > |
Thanks for the feedback :)
My idea when I wrote that was that you can chain keyBy statements to maintain order if your key does not change. Otherwise you are right, we need a sorting operator. Gyula Aljoscha Krettek <[hidden email]> ezt írta (időpont: 2015. júl. 30., Cs, 13:18): > Hi, > sorry for the long wait but I finally found the time to read it. It looks > good but the later parts of course still need to be fleshed out. > > I have one comments/questions: > In the description of partitioned state you have this sentence: "Operations > using partitioned state can also benefit from the partial ordering > guarantees that the flink runtime provides, to implement deterministic > behaviour." How do we provide the ordering guarantees. I would assume that > after a keyBy() the ordering in each partition is arbitrary, unless we add > some sorting operator. > > Cheers, > Aljoscha > > On Thu, 23 Jul 2015 at 15:55 Gyula Fóra <[hidden email]> wrote: > > > Hey! > > I started putting together a guide/design document for the streaming > > operator state interfaces and implementations. The idea would be to > create > > a doc that contains all the details about the implementations so anyone > can > > use it as a reference later. > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing > > > > It will probably take me a couple of days to finish it, but in any case > > feel free to comment. > > > > Cheers, > > Gyula > > > |
Ah ok. I think that keyBy() can normally not be chained because we don't
know how the fields in the emitted object change. On Thu, 30 Jul 2015 at 13:40 Gyula Fóra <[hidden email]> wrote: > Thanks for the feedback :) > > My idea when I wrote that was that you can chain keyBy statements to > maintain order if your key does not change. Otherwise you are right, we > need a sorting operator. > > Gyula > > Aljoscha Krettek <[hidden email]> ezt írta (időpont: 2015. júl. 30., > Cs, 13:18): > > > Hi, > > sorry for the long wait but I finally found the time to read it. It looks > > good but the later parts of course still need to be fleshed out. > > > > I have one comments/questions: > > In the description of partitioned state you have this sentence: > "Operations > > using partitioned state can also benefit from the partial ordering > > guarantees that the flink runtime provides, to implement deterministic > > behaviour." How do we provide the ordering guarantees. I would assume > that > > after a keyBy() the ordering in each partition is arbitrary, unless we > add > > some sorting operator. > > > > Cheers, > > Aljoscha > > > > On Thu, 23 Jul 2015 at 15:55 Gyula Fóra <[hidden email]> wrote: > > > > > Hey! > > > I started putting together a guide/design document for the streaming > > > operator state interfaces and implementations. The idea would be to > > create > > > a doc that contains all the details about the implementations so anyone > > can > > > use it as a reference later. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing > > > > > > It will probably take me a couple of days to finish it, but in any case > > > feel free to comment. > > > > > > Cheers, > > > Gyula > > > > > > |
Free forum by Nabble | Edit this page |