Hello Flink, Calcite, and Beam dev lists!
Linked below is the second document I promised way back in April regarding a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies for the delay; I thought I was nearly done a while back and then temporal joins expanded to something much larger than expected). To repeat what it says in the doc, my hope is that it can serve various purposes over it's lifetime: - - A discussion ground for ironing out any remaining features necessary for supporting robust streaming semantics in Calcite SQL. - A rough, high-level source of truth for tracking efforts underway in support of this, currently spanning the Calcite, Flink, and Beam projects. - A written specification of the changes that were made, for the sake of understanding the delta after the fact. The first and third points are, IMO, the most important. AFAIK, there are a few features missing still that need to be defined (e.g., triggers equivalents via EMIT, robust temporal join support). I'm also proposing a clear distinction of streams and tables, which I think is important, but which I believe is not the approach most folks have been taking in this area. Sorting out these open issues and then having a concise record of the solutions adopted will be important for providing a solid streaming experience and teaching folks how to use it. At any rate, I would much appreciate it if anyone with an interest in this stuff could please take a look and add comments/suggestions/references to related work in flight/etc as appropriate. For now please use comments/suggestions, but if you really want to dive in with edit access, let me know. The doc: http://s.apache.org/streaming-sql-spec -Tyler |
+[hidden email], since I'm told Apex has a Calcite integration as
well. If anyone on the Apex side wants to join in on the fun, your input would be welcomed! -Tyler On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> wrote: > Hello Flink, Calcite, and Beam dev lists! > > Linked below is the second document I promised way back in April regarding > a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies > for the delay; I thought I was nearly done a while back and then temporal > joins expanded to something much larger than expected). > > To repeat what it says in the doc, my hope is that it can serve various > purposes over it's lifetime: > > - > - A discussion ground for ironing out any remaining features necessary > for supporting robust streaming semantics in Calcite SQL. > > - A rough, high-level source of truth for tracking efforts underway in > support of this, currently spanning the Calcite, Flink, and Beam projects. > > - A written specification of the changes that were made, for the sake > of understanding the delta after the fact. > > The first and third points are, IMO, the most important. AFAIK, there are > a few features missing still that need to be defined (e.g., triggers > equivalents via EMIT, robust temporal join support). I'm also proposing a > clear distinction of streams and tables, which I think is important, but > which I believe is not the approach most folks have been taking in this > area. Sorting out these open issues and then having a concise record of the > solutions adopted will be important for providing a solid streaming > experience and teaching folks how to use it. > > At any rate, I would much appreciate it if anyone with an interest in this > stuff could please take a look and add comments/suggestions/references to > related work in flight/etc as appropriate. For now please use > comments/suggestions, but if you really want to dive in with edit access, > let me know. > > The doc: http://s.apache.org/streaming-sql-spec > > -Tyler > > > |
Thanks for the invitation Tyler. I am sure folks who worked on the calcite
integration and others would be interested. On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <[hidden email]> wrote: > +[hidden email], since I'm told Apex has a Calcite integration as > well. If anyone on the Apex side wants to join in on the fun, your input > would be welcomed! > > -Tyler > > > On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> wrote: > > > Hello Flink, Calcite, and Beam dev lists! > > > > Linked below is the second document I promised way back in April > regarding > > a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies > > for the delay; I thought I was nearly done a while back and then temporal > > joins expanded to something much larger than expected). > > > > To repeat what it says in the doc, my hope is that it can serve various > > purposes over it's lifetime: > > > > - > > - A discussion ground for ironing out any remaining features necessary > > for supporting robust streaming semantics in Calcite SQL. > > > > - A rough, high-level source of truth for tracking efforts underway in > > support of this, currently spanning the Calcite, Flink, and Beam > projects. > > > > - A written specification of the changes that were made, for the sake > > of understanding the delta after the fact. > > > > The first and third points are, IMO, the most important. AFAIK, there are > > a few features missing still that need to be defined (e.g., triggers > > equivalents via EMIT, robust temporal join support). I'm also proposing a > > clear distinction of streams and tables, which I think is important, but > > which I believe is not the approach most folks have been taking in this > > area. Sorting out these open issues and then having a concise record of > the > > solutions adopted will be important for providing a solid streaming > > experience and teaching folks how to use it. > > > > At any rate, I would much appreciate it if anyone with an interest in > this > > stuff could please take a look and add comments/suggestions/references > to > > related work in flight/etc as appropriate. For now please use > > comments/suggestions, but if you really want to dive in with edit access, > > let me know. > > > > The doc: http://s.apache.org/streaming-sql-spec > > > > -Tyler > > > > > > > |
Tyler,
Thanks for this. I am reading the document thoroughly and will give my feedback in a day or two. Julian > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <[hidden email]> wrote: > > Thanks for the invitation Tyler. I am sure folks who worked on the calcite > integration and others would be interested. > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau <[hidden email]> > wrote: > >> +[hidden email], since I'm told Apex has a Calcite integration as >> well. If anyone on the Apex side wants to join in on the fun, your input >> would be welcomed! >> >> -Tyler >> >> >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> wrote: >> >>> Hello Flink, Calcite, and Beam dev lists! >>> >>> Linked below is the second document I promised way back in April >> regarding >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies >>> for the delay; I thought I was nearly done a while back and then temporal >>> joins expanded to something much larger than expected). >>> >>> To repeat what it says in the doc, my hope is that it can serve various >>> purposes over it's lifetime: >>> >>> - >>> - A discussion ground for ironing out any remaining features necessary >>> for supporting robust streaming semantics in Calcite SQL. >>> >>> - A rough, high-level source of truth for tracking efforts underway in >>> support of this, currently spanning the Calcite, Flink, and Beam >> projects. >>> >>> - A written specification of the changes that were made, for the sake >>> of understanding the delta after the fact. >>> >>> The first and third points are, IMO, the most important. AFAIK, there are >>> a few features missing still that need to be defined (e.g., triggers >>> equivalents via EMIT, robust temporal join support). I'm also proposing a >>> clear distinction of streams and tables, which I think is important, but >>> which I believe is not the approach most folks have been taking in this >>> area. Sorting out these open issues and then having a concise record of >> the >>> solutions adopted will be important for providing a solid streaming >>> experience and teaching folks how to use it. >>> >>> At any rate, I would much appreciate it if anyone with an interest in >> this >>> stuff could please take a look and add comments/suggestions/references >> to >>> related work in flight/etc as appropriate. For now please use >>> comments/suggestions, but if you really want to dive in with edit access, >>> let me know. >>> >>> The doc: http://s.apache.org/streaming-sql-spec >>> >>> -Tyler >>> >>> >>> >> |
Thanks for the great write up!
I think this s very good starting point for a detailed discussion about features, syntax and semantics of streaming SQL. I'll comment on the document in the next days and describe Flink's current status, our approaches (or planned approaches) and ask a couple of questions. Thanks, Fabian 2017-07-28 3:05 GMT+02:00 Julian Hyde <[hidden email]>: > Tyler, > > Thanks for this. I am reading the document thoroughly and will give my > feedback in a day or two. > > Julian > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <[hidden email]> > wrote: > > > > Thanks for the invitation Tyler. I am sure folks who worked on the > calcite > > integration and others would be interested. > > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau > <[hidden email]> > > wrote: > > > >> +[hidden email], since I'm told Apex has a Calcite integration as > >> well. If anyone on the Apex side wants to join in on the fun, your input > >> would be welcomed! > >> > >> -Tyler > >> > >> > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> > wrote: > >> > >>> Hello Flink, Calcite, and Beam dev lists! > >>> > >>> Linked below is the second document I promised way back in April > >> regarding > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& > apologies > >>> for the delay; I thought I was nearly done a while back and then > temporal > >>> joins expanded to something much larger than expected). > >>> > >>> To repeat what it says in the doc, my hope is that it can serve various > >>> purposes over it's lifetime: > >>> > >>> - > >>> - A discussion ground for ironing out any remaining features > necessary > >>> for supporting robust streaming semantics in Calcite SQL. > >>> > >>> - A rough, high-level source of truth for tracking efforts underway > in > >>> support of this, currently spanning the Calcite, Flink, and Beam > >> projects. > >>> > >>> - A written specification of the changes that were made, for the sake > >>> of understanding the delta after the fact. > >>> > >>> The first and third points are, IMO, the most important. AFAIK, there > are > >>> a few features missing still that need to be defined (e.g., triggers > >>> equivalents via EMIT, robust temporal join support). I'm also > proposing a > >>> clear distinction of streams and tables, which I think is important, > but > >>> which I believe is not the approach most folks have been taking in this > >>> area. Sorting out these open issues and then having a concise record of > >> the > >>> solutions adopted will be important for providing a solid streaming > >>> experience and teaching folks how to use it. > >>> > >>> At any rate, I would much appreciate it if anyone with an interest in > >> this > >>> stuff could please take a look and add comments/suggestions/references > >> to > >>> related work in flight/etc as appropriate. For now please use > >>> comments/suggestions, but if you really want to dive in with edit > access, > >>> let me know. > >>> > >>> The doc: http://s.apache.org/streaming-sql-spec > >>> > >>> -Tyler > >>> > >>> > >>> > >> > > |
Hi Tyler,
Thanks for putting all the efforts into a doc. It is really well written and organized. I like the most part. The major concern I have is about the "explicit trigger". I left a few comments towards this and would like to know what the others think about it. Regards, Shaoxuan On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <[hidden email]> wrote: > Thanks for the great write up! > > I think this s very good starting point for a detailed discussion about > features, syntax and semantics of streaming SQL. > I'll comment on the document in the next days and describe Flink's current > status, our approaches (or planned approaches) and ask a couple of > questions. > > Thanks, Fabian > > 2017-07-28 3:05 GMT+02:00 Julian Hyde <[hidden email]>: > > > Tyler, > > > > Thanks for this. I am reading the document thoroughly and will give my > > feedback in a day or two. > > > > Julian > > > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni <[hidden email]> > > wrote: > > > > > > Thanks for the invitation Tyler. I am sure folks who worked on the > > calcite > > > integration and others would be interested. > > > > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau > > <[hidden email]> > > > wrote: > > > > > >> +[hidden email], since I'm told Apex has a Calcite integration > as > > >> well. If anyone on the Apex side wants to join in on the fun, your > input > > >> would be welcomed! > > >> > > >> -Tyler > > >> > > >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> > > wrote: > > >> > > >>> Hello Flink, Calcite, and Beam dev lists! > > >>> > > >>> Linked below is the second document I promised way back in April > > >> regarding > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& > > apologies > > >>> for the delay; I thought I was nearly done a while back and then > > temporal > > >>> joins expanded to something much larger than expected). > > >>> > > >>> To repeat what it says in the doc, my hope is that it can serve > various > > >>> purposes over it's lifetime: > > >>> > > >>> - > > >>> - A discussion ground for ironing out any remaining features > > necessary > > >>> for supporting robust streaming semantics in Calcite SQL. > > >>> > > >>> - A rough, high-level source of truth for tracking efforts underway > > in > > >>> support of this, currently spanning the Calcite, Flink, and Beam > > >> projects. > > >>> > > >>> - A written specification of the changes that were made, for the > sake > > >>> of understanding the delta after the fact. > > >>> > > >>> The first and third points are, IMO, the most important. AFAIK, there > > are > > >>> a few features missing still that need to be defined (e.g., triggers > > >>> equivalents via EMIT, robust temporal join support). I'm also > > proposing a > > >>> clear distinction of streams and tables, which I think is important, > > but > > >>> which I believe is not the approach most folks have been taking in > this > > >>> area. Sorting out these open issues and then having a concise record > of > > >> the > > >>> solutions adopted will be important for providing a solid streaming > > >>> experience and teaching folks how to use it. > > >>> > > >>> At any rate, I would much appreciate it if anyone with an interest in > > >> this > > >>> stuff could please take a look and add comments/suggestions/ > references > > >> to > > >>> related work in flight/etc as appropriate. For now please use > > >>> comments/suggestions, but if you really want to dive in with edit > > access, > > >>> let me know. > > >>> > > >>> The doc: http://s.apache.org/streaming-sql-spec > > >>> > > >>> -Tyler > > >>> > > >>> > > >>> > > >> > > > > > |
As promised, I went of the document and made some comments.
I also added a bit of information about the current SQL support in Flink and its internals. Thanks, Fabian 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <[hidden email]>: > Hi Tyler, > Thanks for putting all the efforts into a doc. It is really well written > and organized. > I like the most part. The major concern I have is about the "explicit > trigger". I left a few comments towards this and would like to know what > the others think about it. > > Regards, > Shaoxuan > > On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <[hidden email]> wrote: > > > Thanks for the great write up! > > > > I think this s very good starting point for a detailed discussion about > > features, syntax and semantics of streaming SQL. > > I'll comment on the document in the next days and describe Flink's > current > > status, our approaches (or planned approaches) and ask a couple of > > questions. > > > > Thanks, Fabian > > > > 2017-07-28 3:05 GMT+02:00 Julian Hyde <[hidden email]>: > > > > > Tyler, > > > > > > Thanks for this. I am reading the document thoroughly and will give my > > > feedback in a day or two. > > > > > > Julian > > > > > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni < > [hidden email]> > > > wrote: > > > > > > > > Thanks for the invitation Tyler. I am sure folks who worked on the > > > calcite > > > > integration and others would be interested. > > > > > > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau > > > <[hidden email]> > > > > wrote: > > > > > > > >> +[hidden email], since I'm told Apex has a Calcite integration > > as > > > >> well. If anyone on the Apex side wants to join in on the fun, your > > input > > > >> would be welcomed! > > > >> > > > >> -Tyler > > > >> > > > >> > > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> > > > wrote: > > > >> > > > >>> Hello Flink, Calcite, and Beam dev lists! > > > >>> > > > >>> Linked below is the second document I promised way back in April > > > >> regarding > > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& > > > apologies > > > >>> for the delay; I thought I was nearly done a while back and then > > > temporal > > > >>> joins expanded to something much larger than expected). > > > >>> > > > >>> To repeat what it says in the doc, my hope is that it can serve > > various > > > >>> purposes over it's lifetime: > > > >>> > > > >>> - > > > >>> - A discussion ground for ironing out any remaining features > > > necessary > > > >>> for supporting robust streaming semantics in Calcite SQL. > > > >>> > > > >>> - A rough, high-level source of truth for tracking efforts > underway > > > in > > > >>> support of this, currently spanning the Calcite, Flink, and Beam > > > >> projects. > > > >>> > > > >>> - A written specification of the changes that were made, for the > > sake > > > >>> of understanding the delta after the fact. > > > >>> > > > >>> The first and third points are, IMO, the most important. AFAIK, > there > > > are > > > >>> a few features missing still that need to be defined (e.g., > triggers > > > >>> equivalents via EMIT, robust temporal join support). I'm also > > > proposing a > > > >>> clear distinction of streams and tables, which I think is > important, > > > but > > > >>> which I believe is not the approach most folks have been taking in > > this > > > >>> area. Sorting out these open issues and then having a concise > record > > of > > > >> the > > > >>> solutions adopted will be important for providing a solid streaming > > > >>> experience and teaching folks how to use it. > > > >>> > > > >>> At any rate, I would much appreciate it if anyone with an interest > in > > > >> this > > > >>> stuff could please take a look and add comments/suggestions/ > > references > > > >> to > > > >>> related work in flight/etc as appropriate. For now please use > > > >>> comments/suggestions, but if you really want to dive in with edit > > > access, > > > >>> let me know. > > > >>> > > > >>> The doc: http://s.apache.org/streaming-sql-spec > > > >>> > > > >>> -Tyler > > > >>> > > > >>> > > > >>> > > > >> > > > > > > > > > |
I have problems with a couple of the axioms: that a SQL object is
either a table or a stream, but not both; and that a query is bounded if and only if it contains no unbounded streams. I don't have problems with other axioms, such that a query is either bounded or unbounded. And I haven't looked in detail at triggering semantics; I don't think there will be major issues, but let's clear up the 2 problems above first. I have added a section "Julian’s thoughts on the fundamentals" to the end of the document. Julian On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <[hidden email]> wrote: > As promised, I went of the document and made some comments. > I also added a bit of information about the current SQL support in Flink > and its internals. > > Thanks, Fabian > > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <[hidden email]>: > >> Hi Tyler, >> Thanks for putting all the efforts into a doc. It is really well written >> and organized. >> I like the most part. The major concern I have is about the "explicit >> trigger". I left a few comments towards this and would like to know what >> the others think about it. >> >> Regards, >> Shaoxuan >> >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <[hidden email]> wrote: >> >> > Thanks for the great write up! >> > >> > I think this s very good starting point for a detailed discussion about >> > features, syntax and semantics of streaming SQL. >> > I'll comment on the document in the next days and describe Flink's >> current >> > status, our approaches (or planned approaches) and ask a couple of >> > questions. >> > >> > Thanks, Fabian >> > >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <[hidden email]>: >> > >> > > Tyler, >> > > >> > > Thanks for this. I am reading the document thoroughly and will give my >> > > feedback in a day or two. >> > > >> > > Julian >> > > >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni < >> [hidden email]> >> > > wrote: >> > > > >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the >> > > calcite >> > > > integration and others would be interested. >> > > > >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau >> > > <[hidden email]> >> > > > wrote: >> > > > >> > > >> +[hidden email], since I'm told Apex has a Calcite integration >> > as >> > > >> well. If anyone on the Apex side wants to join in on the fun, your >> > input >> > > >> would be welcomed! >> > > >> >> > > >> -Tyler >> > > >> >> > > >> >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email]> >> > > wrote: >> > > >> >> > > >>> Hello Flink, Calcite, and Beam dev lists! >> > > >>> >> > > >>> Linked below is the second document I promised way back in April >> > > >> regarding >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& >> > > apologies >> > > >>> for the delay; I thought I was nearly done a while back and then >> > > temporal >> > > >>> joins expanded to something much larger than expected). >> > > >>> >> > > >>> To repeat what it says in the doc, my hope is that it can serve >> > various >> > > >>> purposes over it's lifetime: >> > > >>> >> > > >>> - >> > > >>> - A discussion ground for ironing out any remaining features >> > > necessary >> > > >>> for supporting robust streaming semantics in Calcite SQL. >> > > >>> >> > > >>> - A rough, high-level source of truth for tracking efforts >> underway >> > > in >> > > >>> support of this, currently spanning the Calcite, Flink, and Beam >> > > >> projects. >> > > >>> >> > > >>> - A written specification of the changes that were made, for the >> > sake >> > > >>> of understanding the delta after the fact. >> > > >>> >> > > >>> The first and third points are, IMO, the most important. AFAIK, >> there >> > > are >> > > >>> a few features missing still that need to be defined (e.g., >> triggers >> > > >>> equivalents via EMIT, robust temporal join support). I'm also >> > > proposing a >> > > >>> clear distinction of streams and tables, which I think is >> important, >> > > but >> > > >>> which I believe is not the approach most folks have been taking in >> > this >> > > >>> area. Sorting out these open issues and then having a concise >> record >> > of >> > > >> the >> > > >>> solutions adopted will be important for providing a solid streaming >> > > >>> experience and teaching folks how to use it. >> > > >>> >> > > >>> At any rate, I would much appreciate it if anyone with an interest >> in >> > > >> this >> > > >>> stuff could please take a look and add comments/suggestions/ >> > references >> > > >> to >> > > >>> related work in flight/etc as appropriate. For now please use >> > > >>> comments/suggestions, but if you really want to dive in with edit >> > > access, >> > > >>> let me know. >> > > >>> >> > > >>> The doc: http://s.apache.org/streaming-sql-spec >> > > >>> >> > > >>> -Tyler >> > > >>> >> > > >>> >> > > >>> >> > > >> >> > > >> > > >> > >> |
Thank you all for the comments/input, I appreciate the time you've put into
this. I've responded to a handful of the major ones. There are some more I'd like to respond to, but I'm out of time for tonight, so more tomorrow. -Tyler On Tue, Aug 1, 2017 at 12:24 PM Julian Hyde <[hidden email]> wrote: > I have problems with a couple of the axioms: that a SQL object is > either a table or a stream, but not both; and that a query is bounded > if and only if it contains no unbounded streams. > > I don't have problems with other axioms, such that a query is either > bounded or unbounded. And I haven't looked in detail at triggering > semantics; I don't think there will be major issues, but let's clear > up the 2 problems above first. > > I have added a section "Julian’s thoughts on the fundamentals" to the > end of the document. > > Julian > > > On Tue, Aug 1, 2017 at 6:40 AM, Fabian Hueske <[hidden email]> wrote: > > As promised, I went of the document and made some comments. > > I also added a bit of information about the current SQL support in Flink > > and its internals. > > > > Thanks, Fabian > > > > 2017-07-30 13:22 GMT+02:00 Shaoxuan Wang <[hidden email]>: > > > >> Hi Tyler, > >> Thanks for putting all the efforts into a doc. It is really well written > >> and organized. > >> I like the most part. The major concern I have is about the "explicit > >> trigger". I left a few comments towards this and would like to know what > >> the others think about it. > >> > >> Regards, > >> Shaoxuan > >> > >> On Sun, Jul 30, 2017 at 4:43 PM, Fabian Hueske <[hidden email]> > wrote: > >> > >> > Thanks for the great write up! > >> > > >> > I think this s very good starting point for a detailed discussion > about > >> > features, syntax and semantics of streaming SQL. > >> > I'll comment on the document in the next days and describe Flink's > >> current > >> > status, our approaches (or planned approaches) and ask a couple of > >> > questions. > >> > > >> > Thanks, Fabian > >> > > >> > 2017-07-28 3:05 GMT+02:00 Julian Hyde <[hidden email]>: > >> > > >> > > Tyler, > >> > > > >> > > Thanks for this. I am reading the document thoroughly and will give > my > >> > > feedback in a day or two. > >> > > > >> > > Julian > >> > > > >> > > > On Jul 25, 2017, at 12:54 PM, Pramod Immaneni < > >> [hidden email]> > >> > > wrote: > >> > > > > >> > > > Thanks for the invitation Tyler. I am sure folks who worked on the > >> > > calcite > >> > > > integration and others would be interested. > >> > > > > >> > > > On Tue, Jul 25, 2017 at 12:12 PM, Tyler Akidau > >> > > <[hidden email]> > >> > > > wrote: > >> > > > > >> > > >> +[hidden email], since I'm told Apex has a Calcite > integration > >> > as > >> > > >> well. If anyone on the Apex side wants to join in on the fun, > your > >> > input > >> > > >> would be welcomed! > >> > > >> > >> > > >> -Tyler > >> > > >> > >> > > >> > >> > > >> On Mon, Jul 24, 2017 at 4:34 PM Tyler Akidau <[hidden email] > > > >> > > wrote: > >> > > >> > >> > > >>> Hello Flink, Calcite, and Beam dev lists! > >> > > >>> > >> > > >>> Linked below is the second document I promised way back in April > >> > > >> regarding > >> > > >>> a collaborative spec for streaming SQL in Beam/Calcite/Flink (& > >> > > apologies > >> > > >>> for the delay; I thought I was nearly done a while back and then > >> > > temporal > >> > > >>> joins expanded to something much larger than expected). > >> > > >>> > >> > > >>> To repeat what it says in the doc, my hope is that it can serve > >> > various > >> > > >>> purposes over it's lifetime: > >> > > >>> > >> > > >>> - > >> > > >>> - A discussion ground for ironing out any remaining features > >> > > necessary > >> > > >>> for supporting robust streaming semantics in Calcite SQL. > >> > > >>> > >> > > >>> - A rough, high-level source of truth for tracking efforts > >> underway > >> > > in > >> > > >>> support of this, currently spanning the Calcite, Flink, and > Beam > >> > > >> projects. > >> > > >>> > >> > > >>> - A written specification of the changes that were made, for > the > >> > sake > >> > > >>> of understanding the delta after the fact. > >> > > >>> > >> > > >>> The first and third points are, IMO, the most important. AFAIK, > >> there > >> > > are > >> > > >>> a few features missing still that need to be defined (e.g., > >> triggers > >> > > >>> equivalents via EMIT, robust temporal join support). I'm also > >> > > proposing a > >> > > >>> clear distinction of streams and tables, which I think is > >> important, > >> > > but > >> > > >>> which I believe is not the approach most folks have been taking > in > >> > this > >> > > >>> area. Sorting out these open issues and then having a concise > >> record > >> > of > >> > > >> the > >> > > >>> solutions adopted will be important for providing a solid > streaming > >> > > >>> experience and teaching folks how to use it. > >> > > >>> > >> > > >>> At any rate, I would much appreciate it if anyone with an > interest > >> in > >> > > >> this > >> > > >>> stuff could please take a look and add comments/suggestions/ > >> > references > >> > > >> to > >> > > >>> related work in flight/etc as appropriate. For now please use > >> > > >>> comments/suggestions, but if you really want to dive in with > edit > >> > > access, > >> > > >>> let me know. > >> > > >>> > >> > > >>> The doc: http://s.apache.org/streaming-sql-spec > >> > > >>> > >> > > >>> -Tyler > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >> > >> > > > >> > > > >> > > >> > |
Free forum by Nabble | Edit this page |