[DISCUSS] FLIP-123: DDL and DML compatibility for Hive connector

[Rui Li-2]

Hi devs,

I'd like to start a discussion about bringing DDL & DML compatibility for
Hive connector. The proposal mainly aims to implement more DDLs for Hive
connector and allow users to write Hive syntax when using the Hive dialect.
Hopefully this will make it easier for users to migrate to Flink, with
fewer SQL statements that need to be changed.

Please find more details in the FLIP wiki [1]. Feedbacks and suggestions
are appreciated.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector

--
Cheers,
Rui Li

[Kurt Young]

Hi Rui,

Thanks for bringing up this discussion and it makes sense to me though i
have one comment about the FLIP.
There are a few places in the design document saying some features will not
be supported or not included in
this FLIP, but I don't get what will be supported exactly. I can imagine
other users will also have such confusion.
Could you add a table or a list of syntax which will be supported?

Best,
Kurt


On Wed, Apr 1, 2020 at 4:24 PM Rui Li <[hidden email]> wrote:

> Hi devs,
>
> I'd like to start a discussion about bringing DDL & DML compatibility for
> Hive connector. The proposal mainly aims to implement more DDLs for Hive
> connector and allow users to write Hive syntax when using the Hive dialect.
> Hopefully this will make it easier for users to migrate to Flink, with
> fewer SQL statements that need to be changed.
>
> Please find more details in the FLIP wiki [1]. Feedbacks and suggestions
> are appreciated.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
>
> --
> Cheers,
> Rui Li
>

[Rui Li]

Hi Kurt,

Thanks for the feedback and that's a good idea. I have updated the FLIP and
added tables in the "Limited Scope" section to list the supported and
unsupported features. Please have a look and let me know if that makes
sense. Thanks.

On Wed, Apr 8, 2020 at 2:19 PM Kurt Young <[hidden email]> wrote:

> Hi Rui,
>
> Thanks for bringing up this discussion and it makes sense to me though i
> have one comment about the FLIP.
> There are a few places in the design document saying some features will not
> be supported or not included in
> this FLIP, but I don't get what will be supported exactly. I can imagine
> other users will also have such confusion.
> Could you add a table or a list of syntax which will be supported?
>
> Best,
> Kurt
>
>
> On Wed, Apr 1, 2020 at 4:24 PM Rui Li <[hidden email]> wrote:
>
> > Hi devs,
> >
> > I'd like to start a discussion about bringing DDL & DML compatibility for
> > Hive connector. The proposal mainly aims to implement more DDLs for Hive
> > connector and allow users to write Hive syntax when using the Hive
> dialect.
> > Hopefully this will make it easier for users to migrate to Flink, with
> > fewer SQL statements that need to be changed.
> >
> > Please find more details in the FLIP wiki [1]. Feedbacks and suggestions
> > are appreciated.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> >
> > --
> > Cheers,
> > Rui Li
> >
>

--
Best regards!
Rui Li

[Jingsong Li]

Thanks Rui for diving.

+1 for this proposal.

There are still lots of people who love Hive SQL.
And I have seen some people support HQL on presto. Presto, as a famous
computing engine, also supports ANSI SQL as we do. This is quite different
from HQL.

Do you think we must need import `FlinkHiveSqlParserImpl`? This will bother
planner code, if possible, I think it is better to keep dialect things in
sql-parer.
What do you think?

Best,
Jingsong Lee

On Thu, Apr 9, 2020 at 3:56 PM Rui Li <[hidden email]> wrote:

> Hi Kurt,
>
> Thanks for the feedback and that's a good idea. I have updated the FLIP and
> added tables in the "Limited Scope" section to list the supported and
> unsupported features. Please have a look and let me know if that makes
> sense. Thanks.
>
> On Wed, Apr 8, 2020 at 2:19 PM Kurt Young <[hidden email]> wrote:
>
> > Hi Rui,
> >
> > Thanks for bringing up this discussion and it makes sense to me though i
> > have one comment about the FLIP.
> > There are a few places in the design document saying some features will
> not
> > be supported or not included in
> > this FLIP, but I don't get what will be supported exactly. I can imagine
> > other users will also have such confusion.
> > Could you add a table or a list of syntax which will be supported?
> >
> > Best,
> > Kurt
> >
> >
> > On Wed, Apr 1, 2020 at 4:24 PM Rui Li <[hidden email]> wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to start a discussion about bringing DDL & DML compatibility
> for
> > > Hive connector. The proposal mainly aims to implement more DDLs for
> Hive
> > > connector and allow users to write Hive syntax when using the Hive
> > dialect.
> > > Hopefully this will make it easier for users to migrate to Flink, with
> > > fewer SQL statements that need to be changed.
> > >
> > > Please find more details in the FLIP wiki [1]. Feedbacks and
> suggestions
> > > are appreciated.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> > >
> > > --
> > > Cheers,
> > > Rui Li
> > >
> >
>
>
> --
> Best regards!
> Rui Li
>

--
Best, Jingsong Lee

[Rui Li]

Hi Jingsong,

Thanks for your input. I'd prefer to keep Hive syntax to a separate parser,
because 1) Supporting both Hive and Flink syntax in one parser makes the
grammar complicated and less efficient, e.g. we may need a bigger LOOKAHEAD
value to resolve conflicts. 2) It can be difficult to add new features to a
syntax without interfering with the other one. 3) It'll be more difficult
to turn off Hive syntax when Hive dialect is not used.
The separate parser only requires minimum change in the planner, which I
think is acceptable compared to the benefits it brings us.

On Thu, Apr 9, 2020 at 10:42 PM Jingsong Li <[hidden email]> wrote:

> Thanks Rui for diving.
>
> +1 for this proposal.
>
> There are still lots of people who love Hive SQL.
> And I have seen some people support HQL on presto. Presto, as a famous
> computing engine, also supports ANSI SQL as we do. This is quite different
> from HQL.
>
> Do you think we must need import `FlinkHiveSqlParserImpl`? This will bother
> planner code, if possible, I think it is better to keep dialect things in
> sql-parer.
> What do you think?
>
> Best,
> Jingsong Lee
>
> On Thu, Apr 9, 2020 at 3:56 PM Rui Li <[hidden email]> wrote:
>
> > Hi Kurt,
> >
> > Thanks for the feedback and that's a good idea. I have updated the FLIP
> and
> > added tables in the "Limited Scope" section to list the supported and
> > unsupported features. Please have a look and let me know if that makes
> > sense. Thanks.
> >
> > On Wed, Apr 8, 2020 at 2:19 PM Kurt Young <[hidden email]> wrote:
> >
> > > Hi Rui,
> > >
> > > Thanks for bringing up this discussion and it makes sense to me though
> i
> > > have one comment about the FLIP.
> > > There are a few places in the design document saying some features will
> > not
> > > be supported or not included in
> > > this FLIP, but I don't get what will be supported exactly. I can
> imagine
> > > other users will also have such confusion.
> > > Could you add a table or a list of syntax which will be supported?
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Wed, Apr 1, 2020 at 4:24 PM Rui Li <[hidden email]> wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I'd like to start a discussion about bringing DDL & DML compatibility
> > for
> > > > Hive connector. The proposal mainly aims to implement more DDLs for
> > Hive
> > > > connector and allow users to write Hive syntax when using the Hive
> > > dialect.
> > > > Hopefully this will make it easier for users to migrate to Flink,
> with
> > > > fewer SQL statements that need to be changed.
> > > >
> > > > Please find more details in the FLIP wiki [1]. Feedbacks and
> > suggestions
> > > > are appreciated.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> > > >
> > > > --
> > > > Cheers,
> > > > Rui Li
> > > >
> > >
> >
> >
> > --
> > Best regards!
> > Rui Li
> >
>
>
> --
> Best, Jingsong Lee
>

--
Best regards!
Rui Li

[Jingsong Li]

Hi,

I didn't mean to unify parser for Flink SQL and Hive SQL.
I mean we should not put hive things to planner.
Because planner have set conformance. So the parser already know the
dialect things.

A simple way is implement a Flink SqlParserImplFactory with conformance. We
can limit dialect things to parser module.

Best,
Jingsong Lee

On Fri, Apr 10, 2020 at 4:16 PM Rui Li <[hidden email]> wrote:

> Hi Jingsong,
>
> Thanks for your input. I'd prefer to keep Hive syntax to a separate parser,
> because 1) Supporting both Hive and Flink syntax in one parser makes the
> grammar complicated and less efficient, e.g. we may need a bigger LOOKAHEAD
> value to resolve conflicts. 2) It can be difficult to add new features to a
> syntax without interfering with the other one. 3) It'll be more difficult
> to turn off Hive syntax when Hive dialect is not used.
> The separate parser only requires minimum change in the planner, which I
> think is acceptable compared to the benefits it brings us.
>
> On Thu, Apr 9, 2020 at 10:42 PM Jingsong Li <[hidden email]>
> wrote:
>
> > Thanks Rui for diving.
> >
> > +1 for this proposal.
> >
> > There are still lots of people who love Hive SQL.
> > And I have seen some people support HQL on presto. Presto, as a famous
> > computing engine, also supports ANSI SQL as we do. This is quite
> different
> > from HQL.
> >
> > Do you think we must need import `FlinkHiveSqlParserImpl`? This will
> bother
> > planner code, if possible, I think it is better to keep dialect things in
> > sql-parer.
> > What do you think?
> >
> > Best,
> > Jingsong Lee
> >
> > On Thu, Apr 9, 2020 at 3:56 PM Rui Li <[hidden email]> wrote:
> >
> > > Hi Kurt,
> > >
> > > Thanks for the feedback and that's a good idea. I have updated the FLIP
> > and
> > > added tables in the "Limited Scope" section to list the supported and
> > > unsupported features. Please have a look and let me know if that makes
> > > sense. Thanks.
> > >
> > > On Wed, Apr 8, 2020 at 2:19 PM Kurt Young <[hidden email]> wrote:
> > >
> > > > Hi Rui,
> > > >
> > > > Thanks for bringing up this discussion and it makes sense to me
> though
> > i
> > > > have one comment about the FLIP.
> > > > There are a few places in the design document saying some features
> will
> > > not
> > > > be supported or not included in
> > > > this FLIP, but I don't get what will be supported exactly. I can
> > imagine
> > > > other users will also have such confusion.
> > > > Could you add a table or a list of syntax which will be supported?
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Wed, Apr 1, 2020 at 4:24 PM Rui Li <[hidden email]> wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I'd like to start a discussion about bringing DDL & DML
> compatibility
> > > for
> > > > > Hive connector. The proposal mainly aims to implement more DDLs for
> > > Hive
> > > > > connector and allow users to write Hive syntax when using the Hive
> > > > dialect.
> > > > > Hopefully this will make it easier for users to migrate to Flink,
> > with
> > > > > fewer SQL statements that need to be changed.
> > > > >
> > > > > Please find more details in the FLIP wiki [1]. Feedbacks and
> > > suggestions
> > > > > are appreciated.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> > > > >
> > > > > --
> > > > > Cheers,
> > > > > Rui Li
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards!
> > > Rui Li
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>
>
> --
> Best regards!
> Rui Li
>

--
Best, Jingsong Lee

[Rui Li]

Thanks Jingsong for the explanation and it makes sense. I have updated the
FLIP. Let me know if that reflects what you have in mind.

On Mon, Apr 13, 2020 at 4:52 PM Jingsong Li <[hidden email]> wrote:

> Hi,
>
> I didn't mean to unify parser for Flink SQL and Hive SQL.
> I mean we should not put hive things to planner.
> Because planner have set conformance. So the parser already know the
> dialect things.
>
> A simple way is implement a Flink SqlParserImplFactory with conformance. We
> can limit dialect things to parser module.
>
> Best,
> Jingsong Lee
>
> On Fri, Apr 10, 2020 at 4:16 PM Rui Li <[hidden email]> wrote:
>
> > Hi Jingsong,
> >
> > Thanks for your input. I'd prefer to keep Hive syntax to a separate
> parser,
> > because 1) Supporting both Hive and Flink syntax in one parser makes the
> > grammar complicated and less efficient, e.g. we may need a bigger
> LOOKAHEAD
> > value to resolve conflicts. 2) It can be difficult to add new features
> to a
> > syntax without interfering with the other one. 3) It'll be more difficult
> > to turn off Hive syntax when Hive dialect is not used.
> > The separate parser only requires minimum change in the planner, which I
> > think is acceptable compared to the benefits it brings us.
> >
> > On Thu, Apr 9, 2020 at 10:42 PM Jingsong Li <[hidden email]>
> > wrote:
> >
> > > Thanks Rui for diving.
> > >
> > > +1 for this proposal.
> > >
> > > There are still lots of people who love Hive SQL.
> > > And I have seen some people support HQL on presto. Presto, as a famous
> > > computing engine, also supports ANSI SQL as we do. This is quite
> > different
> > > from HQL.
> > >
> > > Do you think we must need import `FlinkHiveSqlParserImpl`? This will
> > bother
> > > planner code, if possible, I think it is better to keep dialect things
> in
> > > sql-parer.
> > > What do you think?
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > On Thu, Apr 9, 2020 at 3:56 PM Rui Li <[hidden email]> wrote:
> > >
> > > > Hi Kurt,
> > > >
> > > > Thanks for the feedback and that's a good idea. I have updated the
> FLIP
> > > and
> > > > added tables in the "Limited Scope" section to list the supported and
> > > > unsupported features. Please have a look and let me know if that
> makes
> > > > sense. Thanks.
> > > >
> > > > On Wed, Apr 8, 2020 at 2:19 PM Kurt Young <[hidden email]> wrote:
> > > >
> > > > > Hi Rui,
> > > > >
> > > > > Thanks for bringing up this discussion and it makes sense to me
> > though
> > > i
> > > > > have one comment about the FLIP.
> > > > > There are a few places in the design document saying some features
> > will
> > > > not
> > > > > be supported or not included in
> > > > > this FLIP, but I don't get what will be supported exactly. I can
> > > imagine
> > > > > other users will also have such confusion.
> > > > > Could you add a table or a list of syntax which will be supported?
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Wed, Apr 1, 2020 at 4:24 PM Rui Li <[hidden email]> wrote:
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I'd like to start a discussion about bringing DDL & DML
> > compatibility
> > > > for
> > > > > > Hive connector. The proposal mainly aims to implement more DDLs
> for
> > > > Hive
> > > > > > connector and allow users to write Hive syntax when using the
> Hive
> > > > > dialect.
> > > > > > Hopefully this will make it easier for users to migrate to Flink,
> > > with
> > > > > > fewer SQL statements that need to be changed.
> > > > > >
> > > > > > Please find more details in the FLIP wiki [1]. Feedbacks and
> > > > suggestions
> > > > > > are appreciated.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector
> > > > > >
> > > > > > --
> > > > > > Cheers,
> > > > > > Rui Li
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards!
> > > > Rui Li
> > > >
> > >
> > >
> > > --
> > > Best, Jingsong Lee
> > >
> >
> >
> > --
> > Best regards!
> > Rui Li
> >
>
>
> --
> Best, Jingsong Lee
>

--
Best regards!
Rui Li