Hi guys,
I am checking out the queriable state API, and it seems that most of the tooling is already available. However, the queriable state is available just for the streaming API, not at the StreamSQL API level. In principle, as the flink-table is aware of the query semantic and data output type, it should be possible to configure the query compilation to nest queriable state in the process/window functions. Is there any plan in this direction? Best, Stefano |
Hi Stefano,
yes there are plan in this direction. Actually, I already worked on such a QueryableStateTableSink [1] in the past but never finished it because of priority shifts. Would be great if somebody wants to contribute this functionality :) Regards, Timo [1] https://github.com/twalthr/flink/tree/QueryableTableSink Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > Hi guys, > > I am checking out the queriable state API, and it seems that most of the tooling is already available. However, the queriable state is available just for the streaming API, not at the StreamSQL API level. In principle, as the flink-table is aware of the query semantic and data output type, it should be possible to configure the query compilation to nest queriable state in the process/window functions. Is there any plan in this direction? > > Best, > Stefano > |
Hi, Timo, I've been planning on the same thing and would like to contribute
that. On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> wrote: > Hi Stefano, > > yes there are plan in this direction. Actually, I already worked on such > a QueryableStateTableSink [1] in the past but never finished it because > of priority shifts. Would be great if somebody wants to contribute this > functionality :) > > Regards, > Timo > > [1] https://github.com/twalthr/flink/tree/QueryableTableSink > > Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > > Hi guys, > > > > I am checking out the queriable state API, and it seems that most of the > tooling is already available. However, the queriable state is available > just for the streaming API, not at the StreamSQL API level. In principle, > as the flink-table is aware of the query semantic and data output type, it > should be possible to configure the query compilation to nest queriable > state in the process/window functions. Is there any plan in this direction? > > > > Best, > > Stefano > > > > -- Software Engineer, MVAD |
Hi Renjie,
that would be great. There is already a Jira issue for it: https://issues.apache.org/jira/browse/FLINK-6968 Feel free to assign it to yourself. You can reuse parts of my code if you want. But maybe it would make sense to have a little design document first about what we want to support. Regards, Timo Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > Hi, Timo, I've been planning on the same thing and would like to contribute > that. > > On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> wrote: > >> Hi Stefano, >> >> yes there are plan in this direction. Actually, I already worked on such >> a QueryableStateTableSink [1] in the past but never finished it because >> of priority shifts. Would be great if somebody wants to contribute this >> functionality :) >> >> Regards, >> Timo >> >> [1] https://github.com/twalthr/flink/tree/QueryableTableSink >> >> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: >>> Hi guys, >>> >>> I am checking out the queriable state API, and it seems that most of the >> tooling is already available. However, the queriable state is available >> just for the streaming API, not at the StreamSQL API level. In principle, >> as the flink-table is aware of the query semantic and data output type, it >> should be possible to configure the query compilation to nest queriable >> state in the process/window functions. Is there any plan in this direction? >>> Best, >>> Stefano >>> >> -- > Liu, Renjie > Software Engineer, MVAD > |
Hi, Timo:
It seems that I can't assign it to myself. Could you please help to assign that to me? My jira username is liurenjie1024 and my email is [hidden email] On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> wrote: > Hi Renjie, > > that would be great. There is already a Jira issue for it: > https://issues.apache.org/jira/browse/FLINK-6968 > > Feel free to assign it to yourself. You can reuse parts of my code if > you want. But maybe it would make sense to have a little design document > first about what we want to support. > > Regards, > Timo > > > Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > > Hi, Timo, I've been planning on the same thing and would like to > contribute > > that. > > > > On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> wrote: > > > >> Hi Stefano, > >> > >> yes there are plan in this direction. Actually, I already worked on such > >> a QueryableStateTableSink [1] in the past but never finished it because > >> of priority shifts. Would be great if somebody wants to contribute this > >> functionality :) > >> > >> Regards, > >> Timo > >> > >> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > >> > >> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > >>> Hi guys, > >>> > >>> I am checking out the queriable state API, and it seems that most of > the > >> tooling is already available. However, the queriable state is available > >> just for the streaming API, not at the StreamSQL API level. In > principle, > >> as the flink-table is aware of the query semantic and data output type, > it > >> should be possible to configure the query compilation to nest queriable > >> state in the process/window functions. Is there any plan in this > direction? > >>> Best, > >>> Stefano > >>> > >> -- > > Liu, Renjie > > Software Engineer, MVAD > > > > -- Software Engineer, MVAD |
I gave you contributor permissions in Jira. You should be able to assign
it to yourself now. Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > Hi, Timo: > It seems that I can't assign it to myself. Could you please help to assign > that to me? > My jira username is liurenjie1024 and my email is [hidden email] > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> wrote: > >> Hi Renjie, >> >> that would be great. There is already a Jira issue for it: >> https://issues.apache.org/jira/browse/FLINK-6968 >> >> Feel free to assign it to yourself. You can reuse parts of my code if >> you want. But maybe it would make sense to have a little design document >> first about what we want to support. >> >> Regards, >> Timo >> >> >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: >>> Hi, Timo, I've been planning on the same thing and would like to >> contribute >>> that. >>> >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> wrote: >>> >>>> Hi Stefano, >>>> >>>> yes there are plan in this direction. Actually, I already worked on such >>>> a QueryableStateTableSink [1] in the past but never finished it because >>>> of priority shifts. Would be great if somebody wants to contribute this >>>> functionality :) >>>> >>>> Regards, >>>> Timo >>>> >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink >>>> >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: >>>>> Hi guys, >>>>> >>>>> I am checking out the queriable state API, and it seems that most of >> the >>>> tooling is already available. However, the queriable state is available >>>> just for the streaming API, not at the StreamSQL API level. In >> principle, >>>> as the flink-table is aware of the query semantic and data output type, >> it >>>> should be possible to configure the query compilation to nest queriable >>>> state in the process/window functions. Is there any plan in this >> direction? >>>>> Best, >>>>> Stefano >>>>> >>>> -- >>> Liu, Renjie >>> Software Engineer, MVAD >>> >> -- > Liu, Renjie > Software Engineer, MVAD > |
Great, thank you.
I'll start by writing a design doc. On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > I gave you contributor permissions in Jira. You should be able to assign > it to yourself now. > > Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > > Hi, Timo: > > It seems that I can't assign it to myself. Could you please help to > assign > > that to me? > > My jira username is liurenjie1024 and my email is > [hidden email] > > > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> wrote: > > > >> Hi Renjie, > >> > >> that would be great. There is already a Jira issue for it: > >> https://issues.apache.org/jira/browse/FLINK-6968 > >> > >> Feel free to assign it to yourself. You can reuse parts of my code if > >> you want. But maybe it would make sense to have a little design document > >> first about what we want to support. > >> > >> Regards, > >> Timo > >> > >> > >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > >>> Hi, Timo, I've been planning on the same thing and would like to > >> contribute > >>> that. > >>> > >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > wrote: > >>> > >>>> Hi Stefano, > >>>> > >>>> yes there are plan in this direction. Actually, I already worked on > such > >>>> a QueryableStateTableSink [1] in the past but never finished it > because > >>>> of priority shifts. Would be great if somebody wants to contribute > this > >>>> functionality :) > >>>> > >>>> Regards, > >>>> Timo > >>>> > >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > >>>> > >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > >>>>> Hi guys, > >>>>> > >>>>> I am checking out the queriable state API, and it seems that most of > >> the > >>>> tooling is already available. However, the queriable state is > available > >>>> just for the streaming API, not at the StreamSQL API level. In > >> principle, > >>>> as the flink-table is aware of the query semantic and data output > type, > >> it > >>>> should be possible to configure the query compilation to nest > queriable > >>>> state in the process/window functions. Is there any plan in this > >> direction? > >>>>> Best, > >>>>> Stefano > >>>>> > >>>> -- > >>> Liu, Renjie > >>> Software Engineer, MVAD > >>> > >> -- > > Liu, Renjie > > Software Engineer, MVAD > > > > -- Software Engineer, MVAD |
Hi Timo, Renjie,
What I was thinking did not include the QueryableStateTableSink, but rather tap in directly into the state of a streaming operator. Perhaps it is the same thing, but just it sounds not intuitive to consider it a sink. So, we would need a way to configure the environment for the query to share the "state name" before the query is executed, and then use this to create the hook for the queriable state in the operator. Perhaps extend the current codegen and operator implementations to get as a parameter the StateDescriptor to be inquired. Looking forward for the design document, will be happy to give you feedback. Best, Stefano -----Original Message----- From: Renjie Liu [mailto:[hidden email]] Sent: Friday, March 02, 2018 11:42 AM To: [hidden email] Subject: Re: StreamSQL queriable state Great, thank you. I'll start by writing a design doc. On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > I gave you contributor permissions in Jira. You should be able to > assign it to yourself now. > > Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > > Hi, Timo: > > It seems that I can't assign it to myself. Could you please help to > assign > > that to me? > > My jira username is liurenjie1024 and my email is > [hidden email] > > > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> wrote: > > > >> Hi Renjie, > >> > >> that would be great. There is already a Jira issue for it: > >> https://issues.apache.org/jira/browse/FLINK-6968 > >> > >> Feel free to assign it to yourself. You can reuse parts of my code > >> if you want. But maybe it would make sense to have a little design > >> document first about what we want to support. > >> > >> Regards, > >> Timo > >> > >> > >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > >>> Hi, Timo, I've been planning on the same thing and would like to > >> contribute > >>> that. > >>> > >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > wrote: > >>> > >>>> Hi Stefano, > >>>> > >>>> yes there are plan in this direction. Actually, I already worked > >>>> on > such > >>>> a QueryableStateTableSink [1] in the past but never finished it > because > >>>> of priority shifts. Would be great if somebody wants to > >>>> contribute > this > >>>> functionality :) > >>>> > >>>> Regards, > >>>> Timo > >>>> > >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > >>>> > >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > >>>>> Hi guys, > >>>>> > >>>>> I am checking out the queriable state API, and it seems that > >>>>> most of > >> the > >>>> tooling is already available. However, the queriable state is > available > >>>> just for the streaming API, not at the StreamSQL API level. In > >> principle, > >>>> as the flink-table is aware of the query semantic and data output > type, > >> it > >>>> should be possible to configure the query compilation to nest > queriable > >>>> state in the process/window functions. Is there any plan in this > >> direction? > >>>>> Best, > >>>>> Stefano > >>>>> > >>>> -- > >>> Liu, Renjie > >>> Software Engineer, MVAD > >>> > >> -- > > Liu, Renjie > > Software Engineer, MVAD > > > > -- Software Engineer, MVAD |
Hi, Timo:
I've read your QueryableStateTableSink implementation and that basically implementes what I want to do. I also want to extend SQL client so that user can do point query against the table sink. Do we still need a design doc for that? It seems that I just need to finish the left part and do some test against it. Hi, Stefano: Your requirement needs some changes to the flink table implementation but I don't know why you need that? For debugging? The operator state is internal and subject to optisimation logic, so I think it maybe meanless to expose that. On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli <[hidden email]> wrote: > Hi Timo, Renjie, > > What I was thinking did not include the QueryableStateTableSink, but > rather tap in directly into the state of a streaming operator. Perhaps it > is the same thing, but just it sounds not intuitive to consider it a sink. > > So, we would need a way to configure the environment for the query to > share the "state name" before the query is executed, and then use this to > create the hook for the queriable state in the operator. Perhaps extend the > current codegen and operator implementations to get as a parameter the > StateDescriptor to be inquired. > > Looking forward for the design document, will be happy to give you > feedback. > > Best, > Stefano > > -----Original Message----- > From: Renjie Liu [mailto:[hidden email]] > Sent: Friday, March 02, 2018 11:42 AM > To: [hidden email] > Subject: Re: StreamSQL queriable state > > Great, thank you. > I'll start by writing a design doc. > > On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > > > I gave you contributor permissions in Jira. You should be able to > > assign it to yourself now. > > > > Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > > > Hi, Timo: > > > It seems that I can't assign it to myself. Could you please help to > > assign > > > that to me? > > > My jira username is liurenjie1024 and my email is > > [hidden email] > > > > > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> > wrote: > > > > > >> Hi Renjie, > > >> > > >> that would be great. There is already a Jira issue for it: > > >> https://issues.apache.org/jira/browse/FLINK-6968 > > >> > > >> Feel free to assign it to yourself. You can reuse parts of my code > > >> if you want. But maybe it would make sense to have a little design > > >> document first about what we want to support. > > >> > > >> Regards, > > >> Timo > > >> > > >> > > >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > > >>> Hi, Timo, I've been planning on the same thing and would like to > > >> contribute > > >>> that. > > >>> > > >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > > wrote: > > >>> > > >>>> Hi Stefano, > > >>>> > > >>>> yes there are plan in this direction. Actually, I already worked > > >>>> on > > such > > >>>> a QueryableStateTableSink [1] in the past but never finished it > > because > > >>>> of priority shifts. Would be great if somebody wants to > > >>>> contribute > > this > > >>>> functionality :) > > >>>> > > >>>> Regards, > > >>>> Timo > > >>>> > > >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > > >>>> > > >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > > >>>>> Hi guys, > > >>>>> > > >>>>> I am checking out the queriable state API, and it seems that > > >>>>> most of > > >> the > > >>>> tooling is already available. However, the queriable state is > > available > > >>>> just for the streaming API, not at the StreamSQL API level. In > > >> principle, > > >>>> as the flink-table is aware of the query semantic and data output > > type, > > >> it > > >>>> should be possible to configure the query compilation to nest > > queriable > > >>>> state in the process/window functions. Is there any plan in this > > >> direction? > > >>>>> Best, > > >>>>> Stefano > > >>>>> > > >>>> -- > > >>> Liu, Renjie > > >>> Software Engineer, MVAD > > >>> > > >> -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > Liu, Renjie Software Engineer, MVAD |
Hi Renjie,
it would be great if we can reuse my QueryableStateTableSink implementation. As far as I know, the queryable state interfaces changed a bit in the last months and a new queryable state client as been added. So my branch needs to be rebased to the current APIs. We don't need a big design document but at least some paragraphs how the APIs should look like. How do you want to extend the SQL client for that? We can also introduce the QueryableStateTableSink in one pull request and do the API in a follow-up PR. What do you think? Regards, Timo Am 3/5/18 um 3:52 AM schrieb Renjie Liu: > Hi, Timo: > I've read your QueryableStateTableSink implementation and that basically > implementes what I want to do. I also want to extend SQL client so that > user can do point query against the table sink. Do we still need a design > doc for that? It seems that I just need to finish the left part and do some > test against it. > > Hi, Stefano: > Your requirement needs some changes to the flink table implementation but I > don't know why you need that? For debugging? The operator state is internal > and subject to optisimation logic, so I think it maybe meanless to expose > that. > > On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli <[hidden email]> > wrote: > >> Hi Timo, Renjie, >> >> What I was thinking did not include the QueryableStateTableSink, but >> rather tap in directly into the state of a streaming operator. Perhaps it >> is the same thing, but just it sounds not intuitive to consider it a sink. >> >> So, we would need a way to configure the environment for the query to >> share the "state name" before the query is executed, and then use this to >> create the hook for the queriable state in the operator. Perhaps extend the >> current codegen and operator implementations to get as a parameter the >> StateDescriptor to be inquired. >> >> Looking forward for the design document, will be happy to give you >> feedback. >> >> Best, >> Stefano >> >> -----Original Message----- >> From: Renjie Liu [mailto:[hidden email]] >> Sent: Friday, March 02, 2018 11:42 AM >> To: [hidden email] >> Subject: Re: StreamSQL queriable state >> >> Great, thank you. >> I'll start by writing a design doc. >> >> On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: >> >>> I gave you contributor permissions in Jira. You should be able to >>> assign it to yourself now. >>> >>> Am 3/2/18 um 11:33 AM schrieb Renjie Liu: >>>> Hi, Timo: >>>> It seems that I can't assign it to myself. Could you please help to >>> assign >>>> that to me? >>>> My jira username is liurenjie1024 and my email is >>> [hidden email] >>>> On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> >> wrote: >>>>> Hi Renjie, >>>>> >>>>> that would be great. There is already a Jira issue for it: >>>>> https://issues.apache.org/jira/browse/FLINK-6968 >>>>> >>>>> Feel free to assign it to yourself. You can reuse parts of my code >>>>> if you want. But maybe it would make sense to have a little design >>>>> document first about what we want to support. >>>>> >>>>> Regards, >>>>> Timo >>>>> >>>>> >>>>> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: >>>>>> Hi, Timo, I've been planning on the same thing and would like to >>>>> contribute >>>>>> that. >>>>>> >>>>>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> >>> wrote: >>>>>>> Hi Stefano, >>>>>>> >>>>>>> yes there are plan in this direction. Actually, I already worked >>>>>>> on >>> such >>>>>>> a QueryableStateTableSink [1] in the past but never finished it >>> because >>>>>>> of priority shifts. Would be great if somebody wants to >>>>>>> contribute >>> this >>>>>>> functionality :) >>>>>>> >>>>>>> Regards, >>>>>>> Timo >>>>>>> >>>>>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink >>>>>>> >>>>>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: >>>>>>>> Hi guys, >>>>>>>> >>>>>>>> I am checking out the queriable state API, and it seems that >>>>>>>> most of >>>>> the >>>>>>> tooling is already available. However, the queriable state is >>> available >>>>>>> just for the streaming API, not at the StreamSQL API level. In >>>>> principle, >>>>>>> as the flink-table is aware of the query semantic and data output >>> type, >>>>> it >>>>>>> should be possible to configure the query compilation to nest >>> queriable >>>>>>> state in the process/window functions. Is there any plan in this >>>>> direction? >>>>>>>> Best, >>>>>>>> Stefano >>>>>>>> >>>>>>> -- >>>>>> Liu, Renjie >>>>>> Software Engineer, MVAD >>>>>> >>>>> -- >>>> Liu, Renjie >>>> Software Engineer, MVAD >>>> >>> -- >> Liu, Renjie >> Software Engineer, MVAD >> |
Hi, Timo:
https://docs.google.com/document/d/18XbrFuwzzRzHR84a48j17BIGpTndSS1q-shSgCBEd8I/edit?usp=sharing is a small design doc for the implementation. I agree with you to put SQL client integration in another PR and focus on the sink implementation in this PR. On Mon, Mar 5, 2018 at 9:33 PM Timo Walther <[hidden email]> wrote: > Hi Renjie, > > it would be great if we can reuse my QueryableStateTableSink > implementation. As far as I know, the queryable state interfaces changed > a bit in the last months and a new queryable state client as been added. > So my branch needs to be rebased to the current APIs. > > We don't need a big design document but at least some paragraphs how the > APIs should look like. How do you want to extend the SQL client for > that? We can also introduce the QueryableStateTableSink in one pull > request and do the API in a follow-up PR. > > What do you think? > > Regards, > Timo > > > Am 3/5/18 um 3:52 AM schrieb Renjie Liu: > > Hi, Timo: > > I've read your QueryableStateTableSink implementation and that basically > > implementes what I want to do. I also want to extend SQL client so that > > user can do point query against the table sink. Do we still need a design > > doc for that? It seems that I just need to finish the left part and do > some > > test against it. > > > > Hi, Stefano: > > Your requirement needs some changes to the flink table implementation > but I > > don't know why you need that? For debugging? The operator state is > internal > > and subject to optisimation logic, so I think it maybe meanless to expose > > that. > > > > On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli < > [hidden email]> > > wrote: > > > >> Hi Timo, Renjie, > >> > >> What I was thinking did not include the QueryableStateTableSink, but > >> rather tap in directly into the state of a streaming operator. Perhaps > it > >> is the same thing, but just it sounds not intuitive to consider it a > sink. > >> > >> So, we would need a way to configure the environment for the query to > >> share the "state name" before the query is executed, and then use this > to > >> create the hook for the queriable state in the operator. Perhaps extend > the > >> current codegen and operator implementations to get as a parameter the > >> StateDescriptor to be inquired. > >> > >> Looking forward for the design document, will be happy to give you > >> feedback. > >> > >> Best, > >> Stefano > >> > >> -----Original Message----- > >> From: Renjie Liu [mailto:[hidden email]] > >> Sent: Friday, March 02, 2018 11:42 AM > >> To: [hidden email] > >> Subject: Re: StreamSQL queriable state > >> > >> Great, thank you. > >> I'll start by writing a design doc. > >> > >> On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > >> > >>> I gave you contributor permissions in Jira. You should be able to > >>> assign it to yourself now. > >>> > >>> Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > >>>> Hi, Timo: > >>>> It seems that I can't assign it to myself. Could you please help to > >>> assign > >>>> that to me? > >>>> My jira username is liurenjie1024 and my email is > >>> [hidden email] > >>>> On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> > >> wrote: > >>>>> Hi Renjie, > >>>>> > >>>>> that would be great. There is already a Jira issue for it: > >>>>> https://issues.apache.org/jira/browse/FLINK-6968 > >>>>> > >>>>> Feel free to assign it to yourself. You can reuse parts of my code > >>>>> if you want. But maybe it would make sense to have a little design > >>>>> document first about what we want to support. > >>>>> > >>>>> Regards, > >>>>> Timo > >>>>> > >>>>> > >>>>> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > >>>>>> Hi, Timo, I've been planning on the same thing and would like to > >>>>> contribute > >>>>>> that. > >>>>>> > >>>>>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > >>> wrote: > >>>>>>> Hi Stefano, > >>>>>>> > >>>>>>> yes there are plan in this direction. Actually, I already worked > >>>>>>> on > >>> such > >>>>>>> a QueryableStateTableSink [1] in the past but never finished it > >>> because > >>>>>>> of priority shifts. Would be great if somebody wants to > >>>>>>> contribute > >>> this > >>>>>>> functionality :) > >>>>>>> > >>>>>>> Regards, > >>>>>>> Timo > >>>>>>> > >>>>>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > >>>>>>> > >>>>>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > >>>>>>>> Hi guys, > >>>>>>>> > >>>>>>>> I am checking out the queriable state API, and it seems that > >>>>>>>> most of > >>>>> the > >>>>>>> tooling is already available. However, the queriable state is > >>> available > >>>>>>> just for the streaming API, not at the StreamSQL API level. In > >>>>> principle, > >>>>>>> as the flink-table is aware of the query semantic and data output > >>> type, > >>>>> it > >>>>>>> should be possible to configure the query compilation to nest > >>> queriable > >>>>>>> state in the process/window functions. Is there any plan in this > >>>>> direction? > >>>>>>>> Best, > >>>>>>>> Stefano > >>>>>>>> > >>>>>>> -- > >>>>>> Liu, Renjie > >>>>>> Software Engineer, MVAD > >>>>>> > >>>>> -- > >>>> Liu, Renjie > >>>> Software Engineer, MVAD > >>>> > >>> -- > >> Liu, Renjie > >> Software Engineer, MVAD > >> > > -- Software Engineer, MVAD |
In reply to this post by Renjie Liu
Hi Timo, Renjie,
Well, the idea is that stream processing could become a complex pipeline of multiple queries and sinking data on a separate sink for monitoring does not seem efficient. In fact, pulling state value on demand would allow to monitor the values of different parts of the stream processing pipeline without needing to deal with arrival/output rate also at the monitoring level in a sink. I'm aware it would require some modifications of the level of the table. However, having a separate sink wouldn't duplicate part of the data with respect to the state and output sink? In fact, you would have to keep also these data in a state for stateful processing. As you are going anyway to have a configuration parameter that should be interpreted during the StreamSQL query compilation, perhaps hooking the queryable state as a parameter for the processing functions (and other streaming operators) would be the easier and would not create any overhead (besides the query to the state, of course). My2c. Best, Stefano -----Original Message----- From: Renjie Liu [mailto:[hidden email]] Sent: Monday, March 05, 2018 3:52 AM To: [hidden email] Subject: Re: StreamSQL queriable state Hi, Timo: I've read your QueryableStateTableSink implementation and that basically implementes what I want to do. I also want to extend SQL client so that user can do point query against the table sink. Do we still need a design doc for that? It seems that I just need to finish the left part and do some test against it. Hi, Stefano: Your requirement needs some changes to the flink table implementation but I don't know why you need that? For debugging? The operator state is internal and subject to optisimation logic, so I think it maybe meanless to expose that. On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli <[hidden email]> wrote: > Hi Timo, Renjie, > > What I was thinking did not include the QueryableStateTableSink, but > rather tap in directly into the state of a streaming operator. Perhaps > it is the same thing, but just it sounds not intuitive to consider it a sink. > > So, we would need a way to configure the environment for the query to > share the "state name" before the query is executed, and then use this > to create the hook for the queriable state in the operator. Perhaps > extend the current codegen and operator implementations to get as a > parameter the StateDescriptor to be inquired. > > Looking forward for the design document, will be happy to give you > feedback. > > Best, > Stefano > > -----Original Message----- > From: Renjie Liu [mailto:[hidden email]] > Sent: Friday, March 02, 2018 11:42 AM > To: [hidden email] > Subject: Re: StreamSQL queriable state > > Great, thank you. > I'll start by writing a design doc. > > On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > > > I gave you contributor permissions in Jira. You should be able to > > assign it to yourself now. > > > > Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > > > Hi, Timo: > > > It seems that I can't assign it to myself. Could you please help > > > to > > assign > > > that to me? > > > My jira username is liurenjie1024 and my email is > > [hidden email] > > > > > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> > wrote: > > > > > >> Hi Renjie, > > >> > > >> that would be great. There is already a Jira issue for it: > > >> https://issues.apache.org/jira/browse/FLINK-6968 > > >> > > >> Feel free to assign it to yourself. You can reuse parts of my > > >> code if you want. But maybe it would make sense to have a little > > >> design document first about what we want to support. > > >> > > >> Regards, > > >> Timo > > >> > > >> > > >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > > >>> Hi, Timo, I've been planning on the same thing and would like to > > >> contribute > > >>> that. > > >>> > > >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > > wrote: > > >>> > > >>>> Hi Stefano, > > >>>> > > >>>> yes there are plan in this direction. Actually, I already > > >>>> worked on > > such > > >>>> a QueryableStateTableSink [1] in the past but never finished it > > because > > >>>> of priority shifts. Would be great if somebody wants to > > >>>> contribute > > this > > >>>> functionality :) > > >>>> > > >>>> Regards, > > >>>> Timo > > >>>> > > >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > > >>>> > > >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > > >>>>> Hi guys, > > >>>>> > > >>>>> I am checking out the queriable state API, and it seems that > > >>>>> most of > > >> the > > >>>> tooling is already available. However, the queriable state is > > available > > >>>> just for the streaming API, not at the StreamSQL API level. In > > >> principle, > > >>>> as the flink-table is aware of the query semantic and data > > >>>> output > > type, > > >> it > > >>>> should be possible to configure the query compilation to nest > > queriable > > >>>> state in the process/window functions. Is there any plan in > > >>>> this > > >> direction? > > >>>>> Best, > > >>>>> Stefano > > >>>>> > > >>>> -- > > >>> Liu, Renjie > > >>> Software Engineer, MVAD > > >>> > > >> -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > Liu, Renjie Software Engineer, MVAD |
I don't think it is a good idea to expose the internal state of a query as
queryable state for the following reasons: 1. plan generation: the streaming programs and operators are created based on the optimized plan. A user cannot know which operators a query will run on. Naming the queryable states would be another issue. 2. query internals: an operator should use the most efficient state representation. A user cannot easily know how the state is organized without looking into the internals of the operator. Hence, querying the state would not provide much value because the result will be hard to interpret without knowing the details. 3. backwards compatibility: we need to be able to reimplement operators (which also includes the design of state). If we expose state as queryable, that would not be possible without breaking compatibility. Better optimization might even remove or merge certain operators. Regarding the queryable state table sink: Yes, in the initial design the result might be replicated in the state, but also might not. This depends on the previous operation. We can later still add optimizations that unify replicated state. Best, Fabian 2018-03-08 7:35 GMT-08:00 Stefano Bortoli <[hidden email]>: > Hi Timo, Renjie, > > Well, the idea is that stream processing could become a complex pipeline > of multiple queries and sinking data on a separate sink for monitoring does > not seem efficient. In fact, pulling state value on demand would allow to > monitor the values of different parts of the stream processing pipeline > without needing to deal with arrival/output rate also at the monitoring > level in a sink. > > I'm aware it would require some modifications of the level of the table. > However, having a separate sink wouldn't duplicate part of the data with > respect to the state and output sink? In fact, you would have to keep also > these data in a state for stateful processing. > > As you are going anyway to have a configuration parameter that should be > interpreted during the StreamSQL query compilation, perhaps hooking the > queryable state as a parameter for the processing functions (and other > streaming operators) would be the easier and would not create any overhead > (besides the query to the state, of course). > > My2c. > > Best, > Stefano > > -----Original Message----- > From: Renjie Liu [mailto:[hidden email]] > Sent: Monday, March 05, 2018 3:52 AM > To: [hidden email] > Subject: Re: StreamSQL queriable state > > Hi, Timo: > I've read your QueryableStateTableSink implementation and that basically > implementes what I want to do. I also want to extend SQL client so that > user can do point query against the table sink. Do we still need a design > doc for that? It seems that I just need to finish the left part and do some > test against it. > > Hi, Stefano: > Your requirement needs some changes to the flink table implementation but > I don't know why you need that? For debugging? The operator state is > internal and subject to optisimation logic, so I think it maybe meanless to > expose that. > > On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli <[hidden email] > > > wrote: > > > Hi Timo, Renjie, > > > > What I was thinking did not include the QueryableStateTableSink, but > > rather tap in directly into the state of a streaming operator. Perhaps > > it is the same thing, but just it sounds not intuitive to consider it a > sink. > > > > So, we would need a way to configure the environment for the query to > > share the "state name" before the query is executed, and then use this > > to create the hook for the queriable state in the operator. Perhaps > > extend the current codegen and operator implementations to get as a > > parameter the StateDescriptor to be inquired. > > > > Looking forward for the design document, will be happy to give you > > feedback. > > > > Best, > > Stefano > > > > -----Original Message----- > > From: Renjie Liu [mailto:[hidden email]] > > Sent: Friday, March 02, 2018 11:42 AM > > To: [hidden email] > > Subject: Re: StreamSQL queriable state > > > > Great, thank you. > > I'll start by writing a design doc. > > > > On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > > > > > I gave you contributor permissions in Jira. You should be able to > > > assign it to yourself now. > > > > > > Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > > > > Hi, Timo: > > > > It seems that I can't assign it to myself. Could you please help > > > > to > > > assign > > > > that to me? > > > > My jira username is liurenjie1024 and my email is > > > [hidden email] > > > > > > > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> > > wrote: > > > > > > > >> Hi Renjie, > > > >> > > > >> that would be great. There is already a Jira issue for it: > > > >> https://issues.apache.org/jira/browse/FLINK-6968 > > > >> > > > >> Feel free to assign it to yourself. You can reuse parts of my > > > >> code if you want. But maybe it would make sense to have a little > > > >> design document first about what we want to support. > > > >> > > > >> Regards, > > > >> Timo > > > >> > > > >> > > > >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > > > >>> Hi, Timo, I've been planning on the same thing and would like to > > > >> contribute > > > >>> that. > > > >>> > > > >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > > > wrote: > > > >>> > > > >>>> Hi Stefano, > > > >>>> > > > >>>> yes there are plan in this direction. Actually, I already > > > >>>> worked on > > > such > > > >>>> a QueryableStateTableSink [1] in the past but never finished it > > > because > > > >>>> of priority shifts. Would be great if somebody wants to > > > >>>> contribute > > > this > > > >>>> functionality :) > > > >>>> > > > >>>> Regards, > > > >>>> Timo > > > >>>> > > > >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > > > >>>> > > > >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > > > >>>>> Hi guys, > > > >>>>> > > > >>>>> I am checking out the queriable state API, and it seems that > > > >>>>> most of > > > >> the > > > >>>> tooling is already available. However, the queriable state is > > > available > > > >>>> just for the streaming API, not at the StreamSQL API level. In > > > >> principle, > > > >>>> as the flink-table is aware of the query semantic and data > > > >>>> output > > > type, > > > >> it > > > >>>> should be possible to configure the query compilation to nest > > > queriable > > > >>>> state in the process/window functions. Is there any plan in > > > >>>> this > > > >> direction? > > > >>>>> Best, > > > >>>>> Stefano > > > >>>>> > > > >>>> -- > > > >>> Liu, Renjie > > > >>> Software Engineer, MVAD > > > >>> > > > >> -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- > Liu, Renjie > Software Engineer, MVAD > |
In reply to this post by Stefano Bortoli-2
Hi, Stefano:
I agree with Febian that it's not a good idea to expose internal state as queryable to users for the reasons listed by Fabian. As you pointed out with current QueryableStateSink's design, duplicating internal state is inefficient. But this is limited by the api of table sink since we can only fetch change streams of dynamic table. On Thu, Mar 8, 2018 at 11:35 PM Stefano Bortoli <[hidden email]> wrote: > Hi Timo, Renjie, > > Well, the idea is that stream processing could become a complex pipeline > of multiple queries and sinking data on a separate sink for monitoring does > not seem efficient. In fact, pulling state value on demand would allow to > monitor the values of different parts of the stream processing pipeline > without needing to deal with arrival/output rate also at the monitoring > level in a sink. > > I'm aware it would require some modifications of the level of the table. > However, having a separate sink wouldn't duplicate part of the data with > respect to the state and output sink? In fact, you would have to keep also > these data in a state for stateful processing. > > As you are going anyway to have a configuration parameter that should be > interpreted during the StreamSQL query compilation, perhaps hooking the > queryable state as a parameter for the processing functions (and other > streaming operators) would be the easier and would not create any overhead > (besides the query to the state, of course). > > My2c. > > Best, > Stefano > > -----Original Message----- > From: Renjie Liu [mailto:[hidden email]] > Sent: Monday, March 05, 2018 3:52 AM > To: [hidden email] > Subject: Re: StreamSQL queriable state > > Hi, Timo: > I've read your QueryableStateTableSink implementation and that basically > implementes what I want to do. I also want to extend SQL client so that > user can do point query against the table sink. Do we still need a design > doc for that? It seems that I just need to finish the left part and do some > test against it. > > Hi, Stefano: > Your requirement needs some changes to the flink table implementation but > I don't know why you need that? For debugging? The operator state is > internal and subject to optisimation logic, so I think it maybe meanless to > expose that. > > On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli <[hidden email] > > > wrote: > > > Hi Timo, Renjie, > > > > What I was thinking did not include the QueryableStateTableSink, but > > rather tap in directly into the state of a streaming operator. Perhaps > > it is the same thing, but just it sounds not intuitive to consider it a > sink. > > > > So, we would need a way to configure the environment for the query to > > share the "state name" before the query is executed, and then use this > > to create the hook for the queriable state in the operator. Perhaps > > extend the current codegen and operator implementations to get as a > > parameter the StateDescriptor to be inquired. > > > > Looking forward for the design document, will be happy to give you > > feedback. > > > > Best, > > Stefano > > > > -----Original Message----- > > From: Renjie Liu [mailto:[hidden email]] > > Sent: Friday, March 02, 2018 11:42 AM > > To: [hidden email] > > Subject: Re: StreamSQL queriable state > > > > Great, thank you. > > I'll start by writing a design doc. > > > > On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <[hidden email]> wrote: > > > > > I gave you contributor permissions in Jira. You should be able to > > > assign it to yourself now. > > > > > > Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > > > > Hi, Timo: > > > > It seems that I can't assign it to myself. Could you please help > > > > to > > > assign > > > > that to me? > > > > My jira username is liurenjie1024 and my email is > > > [hidden email] > > > > > > > > On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <[hidden email]> > > wrote: > > > > > > > >> Hi Renjie, > > > >> > > > >> that would be great. There is already a Jira issue for it: > > > >> https://issues.apache.org/jira/browse/FLINK-6968 > > > >> > > > >> Feel free to assign it to yourself. You can reuse parts of my > > > >> code if you want. But maybe it would make sense to have a little > > > >> design document first about what we want to support. > > > >> > > > >> Regards, > > > >> Timo > > > >> > > > >> > > > >> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > > > >>> Hi, Timo, I've been planning on the same thing and would like to > > > >> contribute > > > >>> that. > > > >>> > > > >>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <[hidden email]> > > > wrote: > > > >>> > > > >>>> Hi Stefano, > > > >>>> > > > >>>> yes there are plan in this direction. Actually, I already > > > >>>> worked on > > > such > > > >>>> a QueryableStateTableSink [1] in the past but never finished it > > > because > > > >>>> of priority shifts. Would be great if somebody wants to > > > >>>> contribute > > > this > > > >>>> functionality :) > > > >>>> > > > >>>> Regards, > > > >>>> Timo > > > >>>> > > > >>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > > > >>>> > > > >>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > > > >>>>> Hi guys, > > > >>>>> > > > >>>>> I am checking out the queriable state API, and it seems that > > > >>>>> most of > > > >> the > > > >>>> tooling is already available. However, the queriable state is > > > available > > > >>>> just for the streaming API, not at the StreamSQL API level. In > > > >> principle, > > > >>>> as the flink-table is aware of the query semantic and data > > > >>>> output > > > type, > > > >> it > > > >>>> should be possible to configure the query compilation to nest > > > queriable > > > >>>> state in the process/window functions. Is there any plan in > > > >>>> this > > > >> direction? > > > >>>>> Best, > > > >>>>> Stefano > > > >>>>> > > > >>>> -- > > > >>> Liu, Renjie > > > >>> Software Engineer, MVAD > > > >>> > > > >> -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- > Liu, Renjie > Software Engineer, MVAD > Liu, Renjie Software Engineer, MVAD |
Free forum by Nabble | Edit this page |