Hi all,
I would like to start the vote for FLIP-69[1] which is discussed and reached consensus in the discussion thread[2]. The vote will be open for at least 72 hours. I'll try to close it by 2019-11-08 14:30 UTC, unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement <https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html> Best, Terry Wang |
+1 to the long missing feature in Flink SQL.
On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> wrote: > Hi all, > > I would like to start the vote for FLIP-69[1] which is discussed and > reached consensus in the discussion thread[2]. > > The vote will be open for at least 72 hours. I'll try to close it by > 2019-11-08 14:30 UTC, unless there is an objection or not enough votes. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > < > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > > [2] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > < > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > > Best, > Terry Wang > > > > -- Xuefu Zhang "In Honey We Trust!" |
+1 for the enhancement.
On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: > +1 to the long missing feature in Flink SQL. > > On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> wrote: > > > Hi all, > > > > I would like to start the vote for FLIP-69[1] which is discussed and > > reached consensus in the discussion thread[2]. > > > > The vote will be open for at least 72 hours. I'll try to close it by > > 2019-11-08 14:30 UTC, unless there is an objection or not enough votes. > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > > > > [2] > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > < > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > > > > Best, > > Terry Wang > > > > > > > > > > -- > Xuefu Zhang > > "In Honey We Trust!" > |
Hi Terry,
I went over the FLIP in detail again. The FLIP mostly LGTM. A couple issues: - since we on't plan to support catalog ddl, can you remove them from the FLIP? - I found there are some discrepancies in proposed database and table DDLs. For db ddl, the create db syntax proposes specifying k-v properties following "WITH". However, alter db ddl comes with a keyword "DBPROPERTIES": CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ COMMENT database_comment ] [*WITH *( name=value [, name=value]*)] ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( name=value [, name=value]*) IIUIC, are you borrowing syntax from Hive? Note that Hive's db create ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] The same applies to table ddl. The proposed alter table ddl comes with "SET *PROPERTIES* (...)", however, Flink's existing table create ddl since 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax, both create and alter table ddl comes with "TBLPROPERTIES" [1]. I feel it's better to be consistent among our DDLs. One option is to just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed syntax. [1] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL [2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl On Tue, Nov 5, 2019 at 12:54 PM Peter Huang <[hidden email]> wrote: > +1 for the enhancement. > > On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: > > > +1 to the long missing feature in Flink SQL. > > > > On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> wrote: > > > > > Hi all, > > > > > > I would like to start the vote for FLIP-69[1] which is discussed and > > > reached consensus in the discussion thread[2]. > > > > > > The vote will be open for at least 72 hours. I'll try to close it by > > > 2019-11-08 14:30 UTC, unless there is an objection or not enough votes. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > > < > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > > > > > > [2] > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > > < > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > > > > > > Best, > > > Terry Wang > > > > > > > > > > > > > > > > -- > > Xuefu Zhang > > > > "In Honey We Trust!" > > > |
Hi all,
I want to load checkpoint or savepoint metadata on dev . in this case , I want to debug saved checkpoint metadata. And I knew flink provided a api which is Savepoint.load(env, path), but I can’t find it and can’t use it. Anyone who know about this ? Could you help me ? Thanks very much; |
In reply to this post by bowen.li
Hi Bowen:
Thanks for your feedback. Your opinion convinced me and I just remove the section about catalog create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter DDLs. Open to more comments or votes :) ! Best, Terry Wang > 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: > > Hi Terry, > > I went over the FLIP in detail again. The FLIP mostly LGTM. A couple issues: > > - since we on't plan to support catalog ddl, can you remove them from the > FLIP? > - I found there are some discrepancies in proposed database and table DDLs. > For db ddl, the create db syntax proposes specifying k-v properties > following "WITH". However, alter db ddl comes with a keyword "DBPROPERTIES": > > CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ COMMENT > database_comment ] > [*WITH *( name=value [, name=value]*)] > > > ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( > name=value [, name=value]*) > > > IIUIC, are you borrowing syntax from Hive? Note that Hive's db create > ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF NOT > EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] > > The same applies to table ddl. The proposed alter table ddl comes with > "SET *PROPERTIES* (...)", however, Flink's existing table create ddl since > 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax, > both create and alter table ddl comes with "TBLPROPERTIES" [1]. > > I feel it's better to be consistent among our DDLs. One option is to > just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed syntax. > > [1] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > [2] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl > > On Tue, Nov 5, 2019 at 12:54 PM Peter Huang <[hidden email]> > wrote: > >> +1 for the enhancement. >> >> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: >> >>> +1 to the long missing feature in Flink SQL. >>> >>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> wrote: >>> >>>> Hi all, >>>> >>>> I would like to start the vote for FLIP-69[1] which is discussed and >>>> reached consensus in the discussion thread[2]. >>>> >>>> The vote will be open for at least 72 hours. I'll try to close it by >>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough votes. >>>> >>>> [1] >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >>>> < >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >>>>> >>>> [2] >>>> >>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >>>> < >>>> >>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >>>>> >>>> Best, >>>> Terry Wang >>>> >>>> >>>> >>>> >>> >>> -- >>> Xuefu Zhang >>> >>> "In Honey We Trust!" >>> >> |
In reply to this post by qq
Hi,
Savepoint.load(env, path) is in state processor API library, you should add the following dependency in your project. <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-state-processor-api_2.11</artifactId> <version>1.9.1</version> </dependency> You can see the docuementation for more detailed instructions [1]. Best, Jark [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html On Wed, 6 Nov 2019 at 09:21, qq <[hidden email]> wrote: > Hi all, > I want to load checkpoint or savepoint metadata on dev . in this case , > I want to debug saved checkpoint metadata. And I knew flink provided a api > which is Savepoint.load(env, path), but I can’t find it and can’t use it. > Anyone who know about this ? Could you help me ? Thanks very much; > > |
Btw, user questions should be asked in [hidden email] or [hidden email]. The dev
ML is mainly used to discuss development. Best, Jark On Wed, 6 Nov 2019 at 15:36, Jark Wu <[hidden email]> wrote: > Hi, > > Savepoint.load(env, path) is in state processor API library, you should > add the following dependency in your project. > > <dependency> > <groupId>org.apache.flink</groupId> > <artifactId>flink-state-processor-api_2.11</artifactId> > <version>1.9.1</version> > </dependency> > > > You can see the docuementation for more detailed instructions [1]. > > Best, > Jark > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html > > On Wed, 6 Nov 2019 at 09:21, qq <[hidden email]> wrote: > >> Hi all, >> I want to load checkpoint or savepoint metadata on dev . in this case >> , I want to debug saved checkpoint metadata. And I knew flink provided a >> api which is Savepoint.load(env, path), but I can’t find it and can’t use >> it. Anyone who know about this ? Could you help me ? Thanks very much; >> >> |
In reply to this post by Terry Wang
Thanks.
As Terry and I discussed offline yesterday, we added a new section to explain the detailed implementation plan. +1 (binding) from me. Bowen On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> wrote: > Hi Bowen: > Thanks for your feedback. > Your opinion convinced me and I just remove the section about catalog > create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter > DDLs. > Open to more comments or votes :) ! > > Best, > Terry Wang > > > > > 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: > > > > Hi Terry, > > > > I went over the FLIP in detail again. The FLIP mostly LGTM. A couple > issues: > > > > - since we on't plan to support catalog ddl, can you remove them from the > > FLIP? > > - I found there are some discrepancies in proposed database and table > DDLs. > > For db ddl, the create db syntax proposes specifying k-v properties > > following "WITH". However, alter db ddl comes with a keyword > "DBPROPERTIES": > > > > CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ COMMENT > > database_comment ] > > [*WITH *( name=value [, name=value]*)] > > > > > > ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( > > name=value [, name=value]*) > > > > > > IIUIC, are you borrowing syntax from Hive? Note that Hive's db create > > ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF NOT > > EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] > > > > The same applies to table ddl. The proposed alter table ddl comes with > > "SET *PROPERTIES* (...)", however, Flink's existing table create ddl > since > > 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax, > > both create and alter table ddl comes with "TBLPROPERTIES" [1]. > > > > I feel it's better to be consistent among our DDLs. One option is to > > just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed > syntax. > > > > [1] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > > [2] > > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl > > > > On Tue, Nov 5, 2019 at 12:54 PM Peter Huang <[hidden email]> > > wrote: > > > >> +1 for the enhancement. > >> > >> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: > >> > >>> +1 to the long missing feature in Flink SQL. > >>> > >>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> wrote: > >>> > >>>> Hi all, > >>>> > >>>> I would like to start the vote for FLIP-69[1] which is discussed and > >>>> reached consensus in the discussion thread[2]. > >>>> > >>>> The vote will be open for at least 72 hours. I'll try to close it by > >>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough > votes. > >>>> > >>>> [1] > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >>>> < > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >>>>> > >>>> [2] > >>>> > >>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >>>> < > >>>> > >>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >>>>> > >>>> Best, > >>>> Terry Wang > >>>> > >>>> > >>>> > >>>> > >>> > >>> -- > >>> Xuefu Zhang > >>> > >>> "In Honey We Trust!" > >>> > >> > > |
In reply to this post by Jark Wu-2
Hi,
If you just want to debug, maybe you can do this in UT class in module flink-runtime :) so that you do not need to handle the dependency problem, and access problem. Best, Congxian Jark Wu <[hidden email]> 于2019年11月6日周三 下午3:39写道: > Btw, user questions should be asked in [hidden email] or [hidden email]. The > dev > ML is mainly used to discuss development. > > Best, > Jark > > On Wed, 6 Nov 2019 at 15:36, Jark Wu <[hidden email]> wrote: > > > Hi, > > > > Savepoint.load(env, path) is in state processor API library, you should > > add the following dependency in your project. > > > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-state-processor-api_2.11</artifactId> > > <version>1.9.1</version> > > </dependency> > > > > > > You can see the docuementation for more detailed instructions [1]. > > > > Best, > > Jark > > > > [1]: > > > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html > > > > On Wed, 6 Nov 2019 at 09:21, qq <[hidden email]> wrote: > > > >> Hi all, > >> I want to load checkpoint or savepoint metadata on dev . in this case > >> , I want to debug saved checkpoint metadata. And I knew flink provided a > >> api which is Savepoint.load(env, path), but I can’t find it and can’t > use > >> it. Anyone who know about this ? Could you help me ? Thanks very much; > >> > >> > |
In reply to this post by bowen.li
Thanks Terry for driving this forward.
Got one question about DESCRIBE DATABASE: the results display comment and description of a database. While comment can be specified when a database is created, I don't see how users can specify description of the database? On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> wrote: > Thanks. > > As Terry and I discussed offline yesterday, we added a new section to > explain the detailed implementation plan. > > +1 (binding) from me. > > Bowen > > On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> wrote: > > > Hi Bowen: > > Thanks for your feedback. > > Your opinion convinced me and I just remove the section about catalog > > create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter > > DDLs. > > Open to more comments or votes :) ! > > > > Best, > > Terry Wang > > > > > > > > > 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: > > > > > > Hi Terry, > > > > > > I went over the FLIP in detail again. The FLIP mostly LGTM. A couple > > issues: > > > > > > - since we on't plan to support catalog ddl, can you remove them from > the > > > FLIP? > > > - I found there are some discrepancies in proposed database and table > > DDLs. > > > For db ddl, the create db syntax proposes specifying k-v properties > > > following "WITH". However, alter db ddl comes with a keyword > > "DBPROPERTIES": > > > > > > CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ > COMMENT > > > database_comment ] > > > [*WITH *( name=value [, name=value]*)] > > > > > > > > > ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( > > > name=value [, name=value]*) > > > > > > > > > IIUIC, are you borrowing syntax from Hive? Note that Hive's db > create > > > ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF > NOT > > > EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] > > > > > > The same applies to table ddl. The proposed alter table ddl comes > with > > > "SET *PROPERTIES* (...)", however, Flink's existing table create ddl > > since > > > 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax, > > > both create and alter table ddl comes with "TBLPROPERTIES" [1]. > > > > > > I feel it's better to be consistent among our DDLs. One option is to > > > just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed > > syntax. > > > > > > [1] > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > > > [2] > > > > > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl > > > > > > On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < > [hidden email]> > > > wrote: > > > > > >> +1 for the enhancement. > > >> > > >> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: > > >> > > >>> +1 to the long missing feature in Flink SQL. > > >>> > > >>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> > wrote: > > >>> > > >>>> Hi all, > > >>>> > > >>>> I would like to start the vote for FLIP-69[1] which is discussed and > > >>>> reached consensus in the discussion thread[2]. > > >>>> > > >>>> The vote will be open for at least 72 hours. I'll try to close it by > > >>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough > > votes. > > >>>> > > >>>> [1] > > >>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > >>>> < > > >>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > > >>>>> > > >>>> [2] > > >>>> > > >>> > > >> > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > >>>> < > > >>>> > > >>> > > >> > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > > >>>>> > > >>>> Best, > > >>>> Terry Wang > > >>>> > > >>>> > > >>>> > > >>>> > > >>> > > >>> -- > > >>> Xuefu Zhang > > >>> > > >>> "In Honey We Trust!" > > >>> > > >> > > > > > -- Best regards! Rui Li |
Hi Rui~
Description of the database is obtained from `CatalogDatabase#getDescription()` method, which is implement by CatalogDatebaseImpl. Users don’t need to specify the description. Best, Terry Wang > 2019年11月7日 15:40,Rui Li <[hidden email]> 写道: > > Thanks Terry for driving this forward. > Got one question about DESCRIBE DATABASE: the results display comment and > description of a database. While comment can be specified when a database > is created, I don't see how users can specify description of the database? > > On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> wrote: > >> Thanks. >> >> As Terry and I discussed offline yesterday, we added a new section to >> explain the detailed implementation plan. >> >> +1 (binding) from me. >> >> Bowen >> >> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> wrote: >> >>> Hi Bowen: >>> Thanks for your feedback. >>> Your opinion convinced me and I just remove the section about catalog >>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter >>> DDLs. >>> Open to more comments or votes :) ! >>> >>> Best, >>> Terry Wang >>> >>> >>> >>>> 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: >>>> >>>> Hi Terry, >>>> >>>> I went over the FLIP in detail again. The FLIP mostly LGTM. A couple >>> issues: >>>> >>>> - since we on't plan to support catalog ddl, can you remove them from >> the >>>> FLIP? >>>> - I found there are some discrepancies in proposed database and table >>> DDLs. >>>> For db ddl, the create db syntax proposes specifying k-v properties >>>> following "WITH". However, alter db ddl comes with a keyword >>> "DBPROPERTIES": >>>> >>>> CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ >> COMMENT >>>> database_comment ] >>>> [*WITH *( name=value [, name=value]*)] >>>> >>>> >>>> ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( >>>> name=value [, name=value]*) >>>> >>>> >>>> IIUIC, are you borrowing syntax from Hive? Note that Hive's db >> create >>>> ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF >> NOT >>>> EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] >>>> >>>> The same applies to table ddl. The proposed alter table ddl comes >> with >>>> "SET *PROPERTIES* (...)", however, Flink's existing table create ddl >>> since >>>> 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax, >>>> both create and alter table ddl comes with "TBLPROPERTIES" [1]. >>>> >>>> I feel it's better to be consistent among our DDLs. One option is to >>>> just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed >>> syntax. >>>> >>>> [1] >> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL >>>> [2] >>>> >>> >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl >>>> >>>> On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < >> [hidden email]> >>>> wrote: >>>> >>>>> +1 for the enhancement. >>>>> >>>>> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: >>>>> >>>>>> +1 to the long missing feature in Flink SQL. >>>>>> >>>>>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> >> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I would like to start the vote for FLIP-69[1] which is discussed and >>>>>>> reached consensus in the discussion thread[2]. >>>>>>> >>>>>>> The vote will be open for at least 72 hours. I'll try to close it by >>>>>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough >>> votes. >>>>>>> >>>>>>> [1] >>>>>>> >>>>>> >>>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >>>>>>> < >>>>>>> >>>>>> >>>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >>>>>>>> >>>>>>> [2] >>>>>>> >>>>>> >>>>> >>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >>>>>>> < >>>>>>> >>>>>> >>>>> >>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >>>>>>>> >>>>>>> Best, >>>>>>> Terry Wang >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Xuefu Zhang >>>>>> >>>>>> "In Honey We Trust!" >>>>>> >>>>> >>> >>> >> > > > -- > Best regards! > Rui Li |
I see, thanks for the clarification. In current implementation, it seems
just a duplicate of comment. So I'd prefer not to display it for DESCRIBE DATABASE, because 1) users have no control over the content and 2) it's totally redundant. We can add it in the future when we come up with something more meaningful. What do you think? On Thu, Nov 7, 2019 at 3:54 PM Terry Wang <[hidden email]> wrote: > Hi Rui~ > > Description of the database is obtained from > `CatalogDatabase#getDescription()` method, which is implement by > CatalogDatebaseImpl. Users don’t need to specify the description. > > Best, > Terry Wang > > > > > 2019年11月7日 15:40,Rui Li <[hidden email]> 写道: > > > > Thanks Terry for driving this forward. > > Got one question about DESCRIBE DATABASE: the results display comment and > > description of a database. While comment can be specified when a database > > is created, I don't see how users can specify description of the > database? > > > > On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> wrote: > > > >> Thanks. > >> > >> As Terry and I discussed offline yesterday, we added a new section to > >> explain the detailed implementation plan. > >> > >> +1 (binding) from me. > >> > >> Bowen > >> > >> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> wrote: > >> > >>> Hi Bowen: > >>> Thanks for your feedback. > >>> Your opinion convinced me and I just remove the section about catalog > >>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter > >>> DDLs. > >>> Open to more comments or votes :) ! > >>> > >>> Best, > >>> Terry Wang > >>> > >>> > >>> > >>>> 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: > >>>> > >>>> Hi Terry, > >>>> > >>>> I went over the FLIP in detail again. The FLIP mostly LGTM. A couple > >>> issues: > >>>> > >>>> - since we on't plan to support catalog ddl, can you remove them from > >> the > >>>> FLIP? > >>>> - I found there are some discrepancies in proposed database and table > >>> DDLs. > >>>> For db ddl, the create db syntax proposes specifying k-v properties > >>>> following "WITH". However, alter db ddl comes with a keyword > >>> "DBPROPERTIES": > >>>> > >>>> CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ > >> COMMENT > >>>> database_comment ] > >>>> [*WITH *( name=value [, name=value]*)] > >>>> > >>>> > >>>> ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( > >>>> name=value [, name=value]*) > >>>> > >>>> > >>>> IIUIC, are you borrowing syntax from Hive? Note that Hive's db > >> create > >>>> ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF > >> NOT > >>>> EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] > >>>> > >>>> The same applies to table ddl. The proposed alter table ddl comes > >> with > >>>> "SET *PROPERTIES* (...)", however, Flink's existing table create ddl > >>> since > >>>> 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's > syntax, > >>>> both create and alter table ddl comes with "TBLPROPERTIES" [1]. > >>>> > >>>> I feel it's better to be consistent among our DDLs. One option is to > >>>> just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed > >>> syntax. > >>>> > >>>> [1] > >> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > >>>> [2] > >>>> > >>> > >> > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl > >>>> > >>>> On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < > >> [hidden email]> > >>>> wrote: > >>>> > >>>>> +1 for the enhancement. > >>>>> > >>>>> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: > >>>>> > >>>>>> +1 to the long missing feature in Flink SQL. > >>>>>> > >>>>>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> > >> wrote: > >>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I would like to start the vote for FLIP-69[1] which is discussed > and > >>>>>>> reached consensus in the discussion thread[2]. > >>>>>>> > >>>>>>> The vote will be open for at least 72 hours. I'll try to close it > by > >>>>>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough > >>> votes. > >>>>>>> > >>>>>>> [1] > >>>>>>> > >>>>>> > >>>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >>>>>>> < > >>>>>>> > >>>>>> > >>>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >>>>>>>> > >>>>>>> [2] > >>>>>>> > >>>>>> > >>>>> > >>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >>>>>>> < > >>>>>>> > >>>>>> > >>>>> > >>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >>>>>>>> > >>>>>>> Best, > >>>>>>> Terry Wang > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> Xuefu Zhang > >>>>>> > >>>>>> "In Honey We Trust!" > >>>>>> > >>>>> > >>> > >>> > >> > > > > > > -- > > Best regards! > > Rui Li > > -- Best regards! Rui Li |
Hi Rui~
What you suggested makes sense, remove description and detailed description from `DESCRIBE DATABASE`. Open to more comments and votes :) Best, Terry Wang > 2019年11月7日 17:15,Rui Li <[hidden email]> 写道: > > I see, thanks for the clarification. In current implementation, it seems > just a duplicate of comment. So I'd prefer not to display it for DESCRIBE > DATABASE, because 1) users have no control over the content and 2) it's > totally redundant. We can add it in the future when we come up with > something more meaningful. What do you think? > > On Thu, Nov 7, 2019 at 3:54 PM Terry Wang <[hidden email]> wrote: > >> Hi Rui~ >> >> Description of the database is obtained from >> `CatalogDatabase#getDescription()` method, which is implement by >> CatalogDatebaseImpl. Users don’t need to specify the description. >> >> Best, >> Terry Wang >> >> >> >>> 2019年11月7日 15:40,Rui Li <[hidden email]> 写道: >>> >>> Thanks Terry for driving this forward. >>> Got one question about DESCRIBE DATABASE: the results display comment and >>> description of a database. While comment can be specified when a database >>> is created, I don't see how users can specify description of the >> database? >>> >>> On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> wrote: >>> >>>> Thanks. >>>> >>>> As Terry and I discussed offline yesterday, we added a new section to >>>> explain the detailed implementation plan. >>>> >>>> +1 (binding) from me. >>>> >>>> Bowen >>>> >>>> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> wrote: >>>> >>>>> Hi Bowen: >>>>> Thanks for your feedback. >>>>> Your opinion convinced me and I just remove the section about catalog >>>>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter >>>>> DDLs. >>>>> Open to more comments or votes :) ! >>>>> >>>>> Best, >>>>> Terry Wang >>>>> >>>>> >>>>> >>>>>> 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: >>>>>> >>>>>> Hi Terry, >>>>>> >>>>>> I went over the FLIP in detail again. The FLIP mostly LGTM. A couple >>>>> issues: >>>>>> >>>>>> - since we on't plan to support catalog ddl, can you remove them from >>>> the >>>>>> FLIP? >>>>>> - I found there are some discrepancies in proposed database and table >>>>> DDLs. >>>>>> For db ddl, the create db syntax proposes specifying k-v properties >>>>>> following "WITH". However, alter db ddl comes with a keyword >>>>> "DBPROPERTIES": >>>>>> >>>>>> CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ >>>> COMMENT >>>>>> database_comment ] >>>>>> [*WITH *( name=value [, name=value]*)] >>>>>> >>>>>> >>>>>> ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( >>>>>> name=value [, name=value]*) >>>>>> >>>>>> >>>>>> IIUIC, are you borrowing syntax from Hive? Note that Hive's db >>>> create >>>>>> ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF >>>> NOT >>>>>> EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] >>>>>> >>>>>> The same applies to table ddl. The proposed alter table ddl comes >>>> with >>>>>> "SET *PROPERTIES* (...)", however, Flink's existing table create ddl >>>>> since >>>>>> 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's >> syntax, >>>>>> both create and alter table ddl comes with "TBLPROPERTIES" [1]. >>>>>> >>>>>> I feel it's better to be consistent among our DDLs. One option is to >>>>>> just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed >>>>> syntax. >>>>>> >>>>>> [1] >>>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL >>>>>> [2] >>>>>> >>>>> >>>> >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl >>>>>> >>>>>> On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < >>>> [hidden email]> >>>>>> wrote: >>>>>> >>>>>>> +1 for the enhancement. >>>>>>> >>>>>>> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: >>>>>>> >>>>>>>> +1 to the long missing feature in Flink SQL. >>>>>>>> >>>>>>>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> >>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I would like to start the vote for FLIP-69[1] which is discussed >> and >>>>>>>>> reached consensus in the discussion thread[2]. >>>>>>>>> >>>>>>>>> The vote will be open for at least 72 hours. I'll try to close it >> by >>>>>>>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough >>>>> votes. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >>>>>>>>> < >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >>>>>>>>>> >>>>>>>>> [2] >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >>>>>>>>> < >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >>>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Terry Wang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Xuefu Zhang >>>>>>>> >>>>>>>> "In Honey We Trust!" >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Best regards! >>> Rui Li >> >> > > -- > Best regards! > Rui Li |
Hi all,
I think we should focus to discuss the document in [DISCUSS] thread and keep this vote thread purely for voting. Otherwise, it's hard for others to collect feedbacks for this topic. Best, Kurt On Thu, Nov 7, 2019 at 5:51 PM Terry Wang <[hidden email]> wrote: > Hi Rui~ > What you suggested makes sense, remove description and detailed > description from `DESCRIBE DATABASE`. > Open to more comments and votes :) > > Best, > Terry Wang > > > > > 2019年11月7日 17:15,Rui Li <[hidden email]> 写道: > > > > I see, thanks for the clarification. In current implementation, it seems > > just a duplicate of comment. So I'd prefer not to display it for DESCRIBE > > DATABASE, because 1) users have no control over the content and 2) it's > > totally redundant. We can add it in the future when we come up with > > something more meaningful. What do you think? > > > > On Thu, Nov 7, 2019 at 3:54 PM Terry Wang <[hidden email]> wrote: > > > >> Hi Rui~ > >> > >> Description of the database is obtained from > >> `CatalogDatabase#getDescription()` method, which is implement by > >> CatalogDatebaseImpl. Users don’t need to specify the description. > >> > >> Best, > >> Terry Wang > >> > >> > >> > >>> 2019年11月7日 15:40,Rui Li <[hidden email]> 写道: > >>> > >>> Thanks Terry for driving this forward. > >>> Got one question about DESCRIBE DATABASE: the results display comment > and > >>> description of a database. While comment can be specified when a > database > >>> is created, I don't see how users can specify description of the > >> database? > >>> > >>> On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> wrote: > >>> > >>>> Thanks. > >>>> > >>>> As Terry and I discussed offline yesterday, we added a new section to > >>>> explain the detailed implementation plan. > >>>> > >>>> +1 (binding) from me. > >>>> > >>>> Bowen > >>>> > >>>> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> wrote: > >>>> > >>>>> Hi Bowen: > >>>>> Thanks for your feedback. > >>>>> Your opinion convinced me and I just remove the section about catalog > >>>>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from > alter > >>>>> DDLs. > >>>>> Open to more comments or votes :) ! > >>>>> > >>>>> Best, > >>>>> Terry Wang > >>>>> > >>>>> > >>>>> > >>>>>> 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: > >>>>>> > >>>>>> Hi Terry, > >>>>>> > >>>>>> I went over the FLIP in detail again. The FLIP mostly LGTM. A couple > >>>>> issues: > >>>>>> > >>>>>> - since we on't plan to support catalog ddl, can you remove them > from > >>>> the > >>>>>> FLIP? > >>>>>> - I found there are some discrepancies in proposed database and > table > >>>>> DDLs. > >>>>>> For db ddl, the create db syntax proposes specifying k-v properties > >>>>>> following "WITH". However, alter db ddl comes with a keyword > >>>>> "DBPROPERTIES": > >>>>>> > >>>>>> CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ > >>>> COMMENT > >>>>>> database_comment ] > >>>>>> [*WITH *( name=value [, name=value]*)] > >>>>>> > >>>>>> > >>>>>> ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( > >>>>>> name=value [, name=value]*) > >>>>>> > >>>>>> > >>>>>> IIUIC, are you borrowing syntax from Hive? Note that Hive's db > >>>> create > >>>>>> ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF > >>>> NOT > >>>>>> EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] > >>>>>> > >>>>>> The same applies to table ddl. The proposed alter table ddl comes > >>>> with > >>>>>> "SET *PROPERTIES* (...)", however, Flink's existing table create ddl > >>>>> since > >>>>>> 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's > >> syntax, > >>>>>> both create and alter table ddl comes with "TBLPROPERTIES" [1]. > >>>>>> > >>>>>> I feel it's better to be consistent among our DDLs. One option is to > >>>>>> just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed > >>>>> syntax. > >>>>>> > >>>>>> [1] > >>>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > >>>>>> [2] > >>>>>> > >>>>> > >>>> > >> > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl > >>>>>> > >>>>>> On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < > >>>> [hidden email]> > >>>>>> wrote: > >>>>>> > >>>>>>> +1 for the enhancement. > >>>>>>> > >>>>>>> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> wrote: > >>>>>>> > >>>>>>>> +1 to the long missing feature in Flink SQL. > >>>>>>>> > >>>>>>>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> > >>>> wrote: > >>>>>>>> > >>>>>>>>> Hi all, > >>>>>>>>> > >>>>>>>>> I would like to start the vote for FLIP-69[1] which is discussed > >> and > >>>>>>>>> reached consensus in the discussion thread[2]. > >>>>>>>>> > >>>>>>>>> The vote will be open for at least 72 hours. I'll try to close it > >> by > >>>>>>>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough > >>>>> votes. > >>>>>>>>> > >>>>>>>>> [1] > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >>>>>>>>> < > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >>>>>>>>>> > >>>>>>>>> [2] > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >>>>>>>>> < > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >>>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Terry Wang > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Xuefu Zhang > >>>>>>>> > >>>>>>>> "In Honey We Trust!" > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >>> -- > >>> Best regards! > >>> Rui Li > >> > >> > > > > -- > > Best regards! > > Rui Li > > |
Forgot to vote.. +1 from my side.
Best, Kurt On Fri, Nov 8, 2019 at 11:00 AM Kurt Young <[hidden email]> wrote: > Hi all, > > I think we should focus to discuss the document in [DISCUSS] thread and > keep this vote thread purely for voting. > > Otherwise, it's hard for others to collect feedbacks for this topic. > > Best, > Kurt > > > On Thu, Nov 7, 2019 at 5:51 PM Terry Wang <[hidden email]> wrote: > >> Hi Rui~ >> What you suggested makes sense, remove description and detailed >> description from `DESCRIBE DATABASE`. >> Open to more comments and votes :) >> >> Best, >> Terry Wang >> >> >> >> > 2019年11月7日 17:15,Rui Li <[hidden email]> 写道: >> > >> > I see, thanks for the clarification. In current implementation, it seems >> > just a duplicate of comment. So I'd prefer not to display it for >> DESCRIBE >> > DATABASE, because 1) users have no control over the content and 2) it's >> > totally redundant. We can add it in the future when we come up with >> > something more meaningful. What do you think? >> > >> > On Thu, Nov 7, 2019 at 3:54 PM Terry Wang <[hidden email]> wrote: >> > >> >> Hi Rui~ >> >> >> >> Description of the database is obtained from >> >> `CatalogDatabase#getDescription()` method, which is implement by >> >> CatalogDatebaseImpl. Users don’t need to specify the description. >> >> >> >> Best, >> >> Terry Wang >> >> >> >> >> >> >> >>> 2019年11月7日 15:40,Rui Li <[hidden email]> 写道: >> >>> >> >>> Thanks Terry for driving this forward. >> >>> Got one question about DESCRIBE DATABASE: the results display comment >> and >> >>> description of a database. While comment can be specified when a >> database >> >>> is created, I don't see how users can specify description of the >> >> database? >> >>> >> >>> On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> wrote: >> >>> >> >>>> Thanks. >> >>>> >> >>>> As Terry and I discussed offline yesterday, we added a new section to >> >>>> explain the detailed implementation plan. >> >>>> >> >>>> +1 (binding) from me. >> >>>> >> >>>> Bowen >> >>>> >> >>>> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> >> wrote: >> >>>> >> >>>>> Hi Bowen: >> >>>>> Thanks for your feedback. >> >>>>> Your opinion convinced me and I just remove the section about >> catalog >> >>>>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from >> alter >> >>>>> DDLs. >> >>>>> Open to more comments or votes :) ! >> >>>>> >> >>>>> Best, >> >>>>> Terry Wang >> >>>>> >> >>>>> >> >>>>> >> >>>>>> 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: >> >>>>>> >> >>>>>> Hi Terry, >> >>>>>> >> >>>>>> I went over the FLIP in detail again. The FLIP mostly LGTM. A >> couple >> >>>>> issues: >> >>>>>> >> >>>>>> - since we on't plan to support catalog ddl, can you remove them >> from >> >>>> the >> >>>>>> FLIP? >> >>>>>> - I found there are some discrepancies in proposed database and >> table >> >>>>> DDLs. >> >>>>>> For db ddl, the create db syntax proposes specifying k-v properties >> >>>>>> following "WITH". However, alter db ddl comes with a keyword >> >>>>> "DBPROPERTIES": >> >>>>>> >> >>>>>> CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ >> >>>> COMMENT >> >>>>>> database_comment ] >> >>>>>> [*WITH *( name=value [, name=value]*)] >> >>>>>> >> >>>>>> >> >>>>>> ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* ( >> >>>>>> name=value [, name=value]*) >> >>>>>> >> >>>>>> >> >>>>>> IIUIC, are you borrowing syntax from Hive? Note that Hive's db >> >>>> create >> >>>>>> ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) >> [IF >> >>>> NOT >> >>>>>> EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] >> >>>>>> >> >>>>>> The same applies to table ddl. The proposed alter table ddl comes >> >>>> with >> >>>>>> "SET *PROPERTIES* (...)", however, Flink's existing table create >> ddl >> >>>>> since >> >>>>>> 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's >> >> syntax, >> >>>>>> both create and alter table ddl comes with "TBLPROPERTIES" [1]. >> >>>>>> >> >>>>>> I feel it's better to be consistent among our DDLs. One option is >> to >> >>>>>> just remove the "PROPERTIES" and "DBPROPERTIES" keywords in >> proposed >> >>>>> syntax. >> >>>>>> >> >>>>>> [1] >> >>>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL >> >>>>>> [2] >> >>>>>> >> >>>>> >> >>>> >> >> >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl >> >>>>>> >> >>>>>> On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < >> >>>> [hidden email]> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> +1 for the enhancement. >> >>>>>>> >> >>>>>>> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> >> wrote: >> >>>>>>> >> >>>>>>>> +1 to the long missing feature in Flink SQL. >> >>>>>>>> >> >>>>>>>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> >> >>>> wrote: >> >>>>>>>> >> >>>>>>>>> Hi all, >> >>>>>>>>> >> >>>>>>>>> I would like to start the vote for FLIP-69[1] which is discussed >> >> and >> >>>>>>>>> reached consensus in the discussion thread[2]. >> >>>>>>>>> >> >>>>>>>>> The vote will be open for at least 72 hours. I'll try to close >> it >> >> by >> >>>>>>>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough >> >>>>> votes. >> >>>>>>>>> >> >>>>>>>>> [1] >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >> >>>>>>>>> < >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement >> >>>>>>>>>> >> >>>>>>>>> [2] >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >> >>>>>>>>> < >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html >> >>>>>>>>>> >> >>>>>>>>> Best, >> >>>>>>>>> Terry Wang >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Xuefu Zhang >> >>>>>>>> >> >>>>>>>> "In Honey We Trust!" >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> -- >> >>> Best regards! >> >>> Rui Li >> >> >> >> >> > >> > -- >> > Best regards! >> > Rui Li >> >> |
In reply to this post by Congxian Qiu
Hi all,
Thanks very much. I wants to debug checkpoint with code. Below is my code. Anyway I am sorry I doesn’t understand UT class. def demo(): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment env.setParallelism(1) env.enableCheckpointing(10000) val checkpointConfig = env.getCheckpointConfig checkpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE) checkpointConfig.setMinPauseBetweenCheckpoints(5000) checkpointConfig.setCheckpointTimeout(5000) checkpointConfig.setMaxConcurrentCheckpoints(1) checkpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) val fsStateBackend: StateBackend = new FsStateBackend(STATE_BACKEND) env.setStateBackend(fsStateBackend) env.setRestartStrategy(RestartStrategies.fixedDelayRestart(2, 30000)) //TODO recovery my checkpoint here or run this job from my checkpoint // how to run this job with checkpoint metadata ? use CheckpointCoordinator ?? val dataStream: DataStream[String] = env.addSource(streamSource).name("mysource") dataStream.addSink(new MySQLSink).uid("tesCheckpoint").name("mysink") env.execute() } MySQLSink: class MySQLSink extends RichSinkFunction[String] with CheckpointedFunction { private val bufferSize = 50 private var count: AtomicInteger = _ private var cacheData: ListBuffer[String] = ListBuffer[String]() private var checkpointedState: ListState[(String, ListBuffer[String])] = _ override def open(parameters: Configuration): Unit = { count = new AtomicInteger(0) } override def invoke(jsonData: String, context: SinkFunction.Context[_]): Unit = { val flag = count.getAndIncrement() val end: Long = System.currentTimeMillis() val result = jsonData.substring(0,jsonData.length-1) + ",\"fend\":"+end+"}"; if (flag >= bufferSize) { cacheData += result saveDataList() cacheData.clear() count.set(1) } else { cacheData += result } } def saveDataList(): Unit = { } override def close(): Unit = { super.close() } override def snapshotState(context: FunctionSnapshotContext): Unit = { checkpointedState.clear() val buffer = ListBuffer[(String, ListBuffer[String])](("nlcpTestData", cacheData)) checkpointedState.addAll(buffer.toList.asJava) } override def initializeState(context: FunctionInitializationContext): Unit = { val listStateDesc = new ListStateDescriptor[(String, ListBuffer[String])]("nlcpTestData", TypeInformation.of(new TypeHint[(String, ListBuffer[String])]() {})) val stateStore: OperatorStateStore = context.getOperatorStateStore checkpointedState = stateStore.getListState(listStateDesc) if (context.isRestored) { val data = checkpointedState.get().iterator() while (data.hasNext) { cacheData ++= data.next()._2 } } } } > 在 2019年11月7日,12:03,Congxian Qiu <[hidden email]> 写道: > > Hi, > If you just want to debug, maybe you can do this in UT class in module > flink-runtime :) so that you do not need to handle the dependency problem, > and access problem. > > Best, > Congxian > > > Jark Wu <[hidden email]> 于2019年11月6日周三 下午3:39写道: > >> Btw, user questions should be asked in [hidden email] or [hidden email]. The >> dev >> ML is mainly used to discuss development. >> >> Best, >> Jark >> >> On Wed, 6 Nov 2019 at 15:36, Jark Wu <[hidden email]> wrote: >> >>> Hi, >>> >>> Savepoint.load(env, path) is in state processor API library, you should >>> add the following dependency in your project. >>> >>> <dependency> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-state-processor-api_2.11</artifactId> >>> <version>1.9.1</version> >>> </dependency> >>> >>> >>> You can see the docuementation for more detailed instructions [1]. >>> >>> Best, >>> Jark >>> >>> [1]: >>> >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html >>> >>> On Wed, 6 Nov 2019 at 09:21, qq <[hidden email]> wrote: >>> >>>> Hi all, >>>> I want to load checkpoint or savepoint metadata on dev . in this case >>>> , I want to debug saved checkpoint metadata. And I knew flink provided a >>>> api which is Savepoint.load(env, path), but I can’t find it and can’t >> use >>>> it. Anyone who know about this ? Could you help me ? Thanks very much; >>>> >>>> >> > |
In reply to this post by Congxian Qiu
Hi all,
Thanks very much. I wants to debug checkpoint with code. Below is my code. Anyway I am sorry I doesn’t understand UT class.
|
In reply to this post by Kurt Young
Thanks Terry for driving this. +1
With these features, SQL-CLI could be really used. Best, Jingsong Lee On Fri, Nov 8, 2019 at 2:20 PM Kurt Young <[hidden email]> wrote: > Forgot to vote.. +1 from my side. > > Best, > Kurt > > > On Fri, Nov 8, 2019 at 11:00 AM Kurt Young <[hidden email]> wrote: > > > Hi all, > > > > I think we should focus to discuss the document in [DISCUSS] thread and > > keep this vote thread purely for voting. > > > > Otherwise, it's hard for others to collect feedbacks for this topic. > > > > Best, > > Kurt > > > > > > On Thu, Nov 7, 2019 at 5:51 PM Terry Wang <[hidden email]> wrote: > > > >> Hi Rui~ > >> What you suggested makes sense, remove description and detailed > >> description from `DESCRIBE DATABASE`. > >> Open to more comments and votes :) > >> > >> Best, > >> Terry Wang > >> > >> > >> > >> > 2019年11月7日 17:15,Rui Li <[hidden email]> 写道: > >> > > >> > I see, thanks for the clarification. In current implementation, it > seems > >> > just a duplicate of comment. So I'd prefer not to display it for > >> DESCRIBE > >> > DATABASE, because 1) users have no control over the content and 2) > it's > >> > totally redundant. We can add it in the future when we come up with > >> > something more meaningful. What do you think? > >> > > >> > On Thu, Nov 7, 2019 at 3:54 PM Terry Wang <[hidden email]> wrote: > >> > > >> >> Hi Rui~ > >> >> > >> >> Description of the database is obtained from > >> >> `CatalogDatabase#getDescription()` method, which is implement by > >> >> CatalogDatebaseImpl. Users don’t need to specify the description. > >> >> > >> >> Best, > >> >> Terry Wang > >> >> > >> >> > >> >> > >> >>> 2019年11月7日 15:40,Rui Li <[hidden email]> 写道: > >> >>> > >> >>> Thanks Terry for driving this forward. > >> >>> Got one question about DESCRIBE DATABASE: the results display > comment > >> and > >> >>> description of a database. While comment can be specified when a > >> database > >> >>> is created, I don't see how users can specify description of the > >> >> database? > >> >>> > >> >>> On Thu, Nov 7, 2019 at 4:16 AM Bowen Li <[hidden email]> > wrote: > >> >>> > >> >>>> Thanks. > >> >>>> > >> >>>> As Terry and I discussed offline yesterday, we added a new section > to > >> >>>> explain the detailed implementation plan. > >> >>>> > >> >>>> +1 (binding) from me. > >> >>>> > >> >>>> Bowen > >> >>>> > >> >>>> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang <[hidden email]> > >> wrote: > >> >>>> > >> >>>>> Hi Bowen: > >> >>>>> Thanks for your feedback. > >> >>>>> Your opinion convinced me and I just remove the section about > >> catalog > >> >>>>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from > >> alter > >> >>>>> DDLs. > >> >>>>> Open to more comments or votes :) ! > >> >>>>> > >> >>>>> Best, > >> >>>>> Terry Wang > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>>> 2019年11月6日 07:22,Bowen Li <[hidden email]> 写道: > >> >>>>>> > >> >>>>>> Hi Terry, > >> >>>>>> > >> >>>>>> I went over the FLIP in detail again. The FLIP mostly LGTM. A > >> couple > >> >>>>> issues: > >> >>>>>> > >> >>>>>> - since we on't plan to support catalog ddl, can you remove them > >> from > >> >>>> the > >> >>>>>> FLIP? > >> >>>>>> - I found there are some discrepancies in proposed database and > >> table > >> >>>>> DDLs. > >> >>>>>> For db ddl, the create db syntax proposes specifying k-v > properties > >> >>>>>> following "WITH". However, alter db ddl comes with a keyword > >> >>>>> "DBPROPERTIES": > >> >>>>>> > >> >>>>>> CREATE DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ > >> >>>> COMMENT > >> >>>>>> database_comment ] > >> >>>>>> [*WITH *( name=value [, name=value]*)] > >> >>>>>> > >> >>>>>> > >> >>>>>> ALTER DATABASE [ catalogName.] dataBaseName SET *DBPROPERTIES* > ( > >> >>>>>> name=value [, name=value]*) > >> >>>>>> > >> >>>>>> > >> >>>>>> IIUIC, are you borrowing syntax from Hive? Note that Hive's db > >> >>>> create > >> >>>>>> ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) > >> [IF > >> >>>> NOT > >> >>>>>> EXISTS] database_name ... [*WITH DBPROPERTIES* (k=v, ...)];" [1] > >> >>>>>> > >> >>>>>> The same applies to table ddl. The proposed alter table ddl comes > >> >>>> with > >> >>>>>> "SET *PROPERTIES* (...)", however, Flink's existing table create > >> ddl > >> >>>>> since > >> >>>>>> 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's > >> >> syntax, > >> >>>>>> both create and alter table ddl comes with "TBLPROPERTIES" [1]. > >> >>>>>> > >> >>>>>> I feel it's better to be consistent among our DDLs. One option is > >> to > >> >>>>>> just remove the "PROPERTIES" and "DBPROPERTIES" keywords in > >> proposed > >> >>>>> syntax. > >> >>>>>> > >> >>>>>> [1] > >> >>>> > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > >> >>>>>> [2] > >> >>>>>> > >> >>>>> > >> >>>> > >> >> > >> > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl > >> >>>>>> > >> >>>>>> On Tue, Nov 5, 2019 at 12:54 PM Peter Huang < > >> >>>> [hidden email]> > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>>> +1 for the enhancement. > >> >>>>>>> > >> >>>>>>> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z <[hidden email]> > >> wrote: > >> >>>>>>> > >> >>>>>>>> +1 to the long missing feature in Flink SQL. > >> >>>>>>>> > >> >>>>>>>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang <[hidden email]> > >> >>>> wrote: > >> >>>>>>>> > >> >>>>>>>>> Hi all, > >> >>>>>>>>> > >> >>>>>>>>> I would like to start the vote for FLIP-69[1] which is > discussed > >> >> and > >> >>>>>>>>> reached consensus in the discussion thread[2]. > >> >>>>>>>>> > >> >>>>>>>>> The vote will be open for at least 72 hours. I'll try to close > >> it > >> >> by > >> >>>>>>>>> 2019-11-08 14:30 UTC, unless there is an objection or not > enough > >> >>>>> votes. > >> >>>>>>>>> > >> >>>>>>>>> [1] > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>>>> > >> >>>> > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >> >>>>>>>>> < > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>>>> > >> >>>> > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement > >> >>>>>>>>>> > >> >>>>>>>>> [2] > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>>>> > >> >>>> > >> >> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >> >>>>>>>>> < > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>>>> > >> >>>> > >> >> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html > >> >>>>>>>>>> > >> >>>>>>>>> Best, > >> >>>>>>>>> Terry Wang > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> -- > >> >>>>>>>> Xuefu Zhang > >> >>>>>>>> > >> >>>>>>>> "In Honey We Trust!" > >> >>>>>>>> > >> >>>>>>> > >> >>>>> > >> >>>>> > >> >>>> > >> >>> > >> >>> > >> >>> -- > >> >>> Best regards! > >> >>> Rui Li > >> >> > >> >> > >> > > >> > -- > >> > Best regards! > >> > Rui Li > >> > >> > -- Best, Jingsong Lee |
In reply to this post by Terry Wang
+1(non-binding), nice job, Terry ~
Best, Danny Chan 在 2019年11月5日 +0800 PM10:32,[hidden email],写道: > > +1 to the long missing feature in Flink SQL. |
Free forum by Nabble | Edit this page |