[DISCUSS] Support JSON functions in Flink SQL

[Forward Xu]

Hi everybody,

I'd like to kick off a discussion on Support JSON functions in Flink SQL.

The entire plan is divided into two steps:
1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL, such as
JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.

Would love to hear your thoughts.

[1]
https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp

Best,
ForwardXu

[Danny Chan]

Thanks Xu Forward for bring up this topic, I think the JSON functions are very useful especially for those MySQL users.

I saw that you have done some work within the Apache Calcite, that’s a good start, but this is one concern from me, Flink doesn’t support JSON type internal, so how to represent a JSON object in Flink maybe a key point we need to resolve. In Calcite, we use ANY type to represent as the JSON, but I don’t think it is the right way to go, maybe we can have a discussion here.

Best,
Danny Chan
在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：

> Hi everybody,
>
> I'd like to kick off a discussion on Support JSON functions in Flink SQL.
>
> The entire plan is divided into two steps:
> 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL, such as
> JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
>
> Would love to hear your thoughts.
>
> [1]
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
>
> Best,
> ForwardXu

[Forward Xu]

hi Danny Chan ,Thank you very much for your reply, your help can help me
further improve this discussion.
Best
forward

Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：

> Thanks Xu Forward for bring up this topic, I think the JSON functions are
> very useful especially for those MySQL users.
>
> I saw that you have done some work within the Apache Calcite, that’s a
> good start, but this is one concern from me, Flink doesn’t support JSON
> type internal, so how to represent a JSON object in Flink maybe a key point
> we need to resolve. In Calcite, we use ANY type to represent as the JSON,
> but I don’t think it is the right way to go, maybe we can have a discussion
> here.
>
> Best,
> Danny Chan
> 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > Hi everybody,
> >
> > I'd like to kick off a discussion on Support JSON functions in Flink SQL.
> >
> > The entire plan is divided into two steps:
> > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL, such
> as
> > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> >
> > Would love to hear your thoughts.
> >
> > [1]
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> >
> > Best,
> > ForwardXu
>

[Jark Wu-2]

Hi Forward,

Thanks for bringing this discussion and preparing the nice design.
I think it's nice to have the JSON functions in the next release.
We have received some requirements for this feature.

I can help to shepherd this JSON functions effort and will leave comments
 in the design doc in the next days.

Hi Danny,

The new introduced JSON functions are from SQL:2016, not from MySQL.
So there no JSON type is needed. According to the SQL:2016, the
representation of JSON data can be "character string" which is also
the current implementation in Calcite[1].

Best,
Jark


[1]: https://calcite.apache.org/docs/reference.html#json-functions


On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]> wrote:

> hi Danny Chan ,Thank you very much for your reply, your help can help me
> further improve this discussion.
> Best
> forward
>
> Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
>
> > Thanks Xu Forward for bring up this topic, I think the JSON functions are
> > very useful especially for those MySQL users.
> >
> > I saw that you have done some work within the Apache Calcite, that’s a
> > good start, but this is one concern from me, Flink doesn’t support JSON
> > type internal, so how to represent a JSON object in Flink maybe a key
> point
> > we need to resolve. In Calcite, we use ANY type to represent as the JSON,
> > but I don’t think it is the right way to go, maybe we can have a
> discussion
> > here.
> >
> > Best,
> > Danny Chan
> > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > Hi everybody,
> > >
> > > I'd like to kick off a discussion on Support JSON functions in Flink
> SQL.
> > >
> > > The entire plan is divided into two steps:
> > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL,
> such
> > as
> > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> > >
> > > Would love to hear your thoughts.
> > >
> > > [1]
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > >
> > > Best,
> > > ForwardXu
> >
>

[qi luo]

We also see strong demands from our SQL users for JSON/Date related
functions.

Also +Anyang Hu <[hidden email]>

On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:

> Hi Forward,
>
> Thanks for bringing this discussion and preparing the nice design.
> I think it's nice to have the JSON functions in the next release.
> We have received some requirements for this feature.
>
> I can help to shepherd this JSON functions effort and will leave comments
>  in the design doc in the next days.
>
> Hi Danny,
>
> The new introduced JSON functions are from SQL:2016, not from MySQL.
> So there no JSON type is needed. According to the SQL:2016, the
> representation of JSON data can be "character string" which is also
> the current implementation in Calcite[1].
>
> Best,
> Jark
>
>
> [1]: https://calcite.apache.org/docs/reference.html#json-functions
>
>
> On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]> wrote:
>
> > hi Danny Chan ,Thank you very much for your reply, your help can help me
> > further improve this discussion.
> > Best
> > forward
> >
> > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> >
> > > Thanks Xu Forward for bring up this topic, I think the JSON functions
> are
> > > very useful especially for those MySQL users.
> > >
> > > I saw that you have done some work within the Apache Calcite, that’s a
> > > good start, but this is one concern from me, Flink doesn’t support JSON
> > > type internal, so how to represent a JSON object in Flink maybe a key
> > point
> > > we need to resolve. In Calcite, we use ANY type to represent as the
> JSON,
> > > but I don’t think it is the right way to go, maybe we can have a
> > discussion
> > > here.
> > >
> > > Best,
> > > Danny Chan
> > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > > Hi everybody,
> > > >
> > > > I'd like to kick off a discussion on Support JSON functions in Flink
> > SQL.
> > > >
> > > > The entire plan is divided into two steps:
> > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL,
> > such
> > > as
> > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> > > >
> > > > Would love to hear your thoughts.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > >
> > > > Best,
> > > > ForwardXu
> > >
> >
>

[Kurt Young]

+1 to add JSON support to Flink. We also see lots of requirements for JSON
related functions in our internal platform. Since these are already SQL
standard, I think it's a good time to add them to Flink.

Best,
Kurt


On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:

> We also see strong demands from our SQL users for JSON/Date related
> functions.
>
> Also +Anyang Hu <[hidden email]>
>
> On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
>
> > Hi Forward,
> >
> > Thanks for bringing this discussion and preparing the nice design.
> > I think it's nice to have the JSON functions in the next release.
> > We have received some requirements for this feature.
> >
> > I can help to shepherd this JSON functions effort and will leave comments
> >  in the design doc in the next days.
> >
> > Hi Danny,
> >
> > The new introduced JSON functions are from SQL:2016, not from MySQL.
> > So there no JSON type is needed. According to the SQL:2016, the
> > representation of JSON data can be "character string" which is also
> > the current implementation in Calcite[1].
> >
> > Best,
> > Jark
> >
> >
> > [1]: https://calcite.apache.org/docs/reference.html#json-functions
> >
> >
> > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]> wrote:
> >
> > > hi Danny Chan ,Thank you very much for your reply, your help can help
> me
> > > further improve this discussion.
> > > Best
> > > forward
> > >
> > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > >
> > > > Thanks Xu Forward for bring up this topic, I think the JSON functions
> > are
> > > > very useful especially for those MySQL users.
> > > >
> > > > I saw that you have done some work within the Apache Calcite, that’s
> a
> > > > good start, but this is one concern from me, Flink doesn’t support
> JSON
> > > > type internal, so how to represent a JSON object in Flink maybe a key
> > > point
> > > > we need to resolve. In Calcite, we use ANY type to represent as the
> > JSON,
> > > > but I don’t think it is the right way to go, maybe we can have a
> > > discussion
> > > > here.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > > > Hi everybody,
> > > > >
> > > > > I'd like to kick off a discussion on Support JSON functions in
> Flink
> > > SQL.
> > > > >
> > > > > The entire plan is divided into two steps:
> > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL,
> > > such
> > > > as
> > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> > > > >
> > > > > Would love to hear your thoughts.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > >
> > > > > Best,
> > > > > ForwardXu
> > > >
> > >
> >
>

+1
I have done similar work before.
Looking forward to discussing this feature.

Best
wenhui



[hidden email]
 
From: Kurt Young
Date: 2019-09-05 14:00
To: dev
CC: Anyang Hu
Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
+1 to add JSON support to Flink. We also see lots of requirements for JSON
related functions in our internal platform. Since these are already SQL
standard, I think it's a good time to add them to Flink.
 
Best,
Kurt
 
 
On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:
 

> We also see strong demands from our SQL users for JSON/Date related
> functions.
>
> Also +Anyang Hu <[hidden email]>
>
> On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
>
> > Hi Forward,
> >
> > Thanks for bringing this discussion and preparing the nice design.
> > I think it's nice to have the JSON functions in the next release.
> > We have received some requirements for this feature.
> >
> > I can help to shepherd this JSON functions effort and will leave comments
> >  in the design doc in the next days.
> >
> > Hi Danny,
> >
> > The new introduced JSON functions are from SQL:2016, not from MySQL.
> > So there no JSON type is needed. According to the SQL:2016, the
> > representation of JSON data can be "character string" which is also
> > the current implementation in Calcite[1].
> >
> > Best,
> > Jark
> >
> >
> > [1]: https://calcite.apache.org/docs/reference.html#json-functions
> >
> >
> > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]> wrote:
> >
> > > hi Danny Chan ,Thank you very much for your reply, your help can help
> me
> > > further improve this discussion.
> > > Best
> > > forward
> > >
> > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > >
> > > > Thanks Xu Forward for bring up this topic, I think the JSON functions
> > are
> > > > very useful especially for those MySQL users.
> > > >
> > > > I saw that you have done some work within the Apache Calcite, that’s
> a
> > > > good start, but this is one concern from me, Flink doesn’t support
> JSON
> > > > type internal, so how to represent a JSON object in Flink maybe a key
> > > point
> > > > we need to resolve. In Calcite, we use ANY type to represent as the
> > JSON,
> > > > but I don’t think it is the right way to go, maybe we can have a
> > > discussion
> > > > here.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > > > Hi everybody,
> > > > >
> > > > > I'd like to kick off a discussion on Support JSON functions in
> Flink
> > > SQL.
> > > > >
> > > > > The entire plan is divided into two steps:
> > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL,
> > > such
> > > > as
> > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> > > > >
> > > > > Would love to hear your thoughts.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > >
> > > > > Best,
> > > > > ForwardXu
> > > >
> > >
> >
>

[JingsongLee-2]

+1
Nice document. I think it is easier to do after expression reworking[1].
By the way, which planner do you want to start?

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design

Best,
Jingsong Lee


------------------------------------------------------------------
From:TANG Wen-hui <[hidden email]>
Send Time:2019年9月5日(星期四) 14:36
To:dev <[hidden email]>
Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL

+1
I have done similar work before.
Looking forward to discussing this feature.

Best
wenhui



[hidden email]

From: Kurt Young
Date: 2019-09-05 14:00
To: dev
CC: Anyang Hu
Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
+1 to add JSON support to Flink. We also see lots of requirements for JSON
related functions in our internal platform. Since these are already SQL
standard, I think it's a good time to add them to Flink.

Best,
Kurt


On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:

> We also see strong demands from our SQL users for JSON/Date related
> functions.
>
> Also +Anyang Hu <[hidden email]>
>
> On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
>
> > Hi Forward,
> >
> > Thanks for bringing this discussion and preparing the nice design.
> > I think it's nice to have the JSON functions in the next release.
> > We have received some requirements for this feature.
> >
> > I can help to shepherd this JSON functions effort and will leave comments
> >  in the design doc in the next days.
> >
> > Hi Danny,
> >
> > The new introduced JSON functions are from SQL:2016, not from MySQL.
> > So there no JSON type is needed. According to the SQL:2016, the
> > representation of JSON data can be "character string" which is also
> > the current implementation in Calcite[1].
> >
> > Best,
> > Jark
> >
> >
> > [1]: https://calcite.apache.org/docs/reference.html#json-functions
> >
> >
> > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]> wrote:
> >
> > > hi Danny Chan ,Thank you very much for your reply, your help can help
> me
> > > further improve this discussion.
> > > Best
> > > forward
> > >
> > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > >
> > > > Thanks Xu Forward for bring up this topic, I think the JSON functions
> > are
> > > > very useful especially for those MySQL users.
> > > >
> > > > I saw that you have done some work within the Apache Calcite, that’s
> a
> > > > good start, but this is one concern from me, Flink doesn’t support
> JSON
> > > > type internal, so how to represent a JSON object in Flink maybe a key
> > > point
> > > > we need to resolve. In Calcite, we use ANY type to represent as the
> > JSON,
> > > > but I don’t think it is the right way to go, maybe we can have a
> > > discussion
> > > > here.
> > > >
> > > > Best,
> > > > Danny Chan
> > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > > > Hi everybody,
> > > > >
> > > > > I'd like to kick off a discussion on Support JSON functions in
> Flink
> > > SQL.
> > > > >
> > > > > The entire plan is divided into two steps:
> > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].
> > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink SQL,
> > > such
> > > > as
> > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> > > > >
> > > > > Would love to hear your thoughts.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > >
> > > > > Best,
> > > > > ForwardXu
> > > >
> > >
> >
>

[vino yang]

+1 to have JSON functions in Flink SQL

JingsongLee <[hidden email]> 于2019年9月5日周四 下午4:46写道：

> +1
> Nice document. I think it is easier to do after expression reworking[1].
> By the way, which planner do you want to start?
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>
> Best,
> Jingsong Lee
>
>
> ------------------------------------------------------------------
> From:TANG Wen-hui <[hidden email]>
> Send Time:2019年9月5日(星期四) 14:36
> To:dev <[hidden email]>
> Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
>
> +1
> I have done similar work before.
> Looking forward to discussing this feature.
>
> Best
> wenhui
>
>
>
> [hidden email]
>
> From: Kurt Young
> Date: 2019-09-05 14:00
> To: dev
> CC: Anyang Hu
> Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> +1 to add JSON support to Flink. We also see lots of requirements for JSON
> related functions in our internal platform. Since these are already SQL
> standard, I think it's a good time to add them to Flink.
>
> Best,
> Kurt
>
>
> On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:
>
> > We also see strong demands from our SQL users for JSON/Date related
> > functions.
> >
> > Also +Anyang Hu <[hidden email]>
> >
> > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
> >
> > > Hi Forward,
> > >
> > > Thanks for bringing this discussion and preparing the nice design.
> > > I think it's nice to have the JSON functions in the next release.
> > > We have received some requirements for this feature.
> > >
> > > I can help to shepherd this JSON functions effort and will leave
> comments
> > >  in the design doc in the next days.
> > >
> > > Hi Danny,
> > >
> > > The new introduced JSON functions are from SQL:2016, not from MySQL.
> > > So there no JSON type is needed. According to the SQL:2016, the
> > > representation of JSON data can be "character string" which is also
> > > the current implementation in Calcite[1].
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > [1]: https://calcite.apache.org/docs/reference.html#json-functions
> > >
> > >
> > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]>
> wrote:
> > >
> > > > hi Danny Chan ,Thank you very much for your reply, your help can help
> > me
> > > > further improve this discussion.
> > > > Best
> > > > forward
> > > >
> > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > > >
> > > > > Thanks Xu Forward for bring up this topic, I think the JSON
> functions
> > > are
> > > > > very useful especially for those MySQL users.
> > > > >
> > > > > I saw that you have done some work within the Apache Calcite,
> that’s
> > a
> > > > > good start, but this is one concern from me, Flink doesn’t support
> > JSON
> > > > > type internal, so how to represent a JSON object in Flink maybe a
> key
> > > > point
> > > > > we need to resolve. In Calcite, we use ANY type to represent as the
> > > JSON,
> > > > > but I don’t think it is the right way to go, maybe we can have a
> > > > discussion
> > > > > here.
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > > > > Hi everybody,
> > > > > >
> > > > > > I'd like to kick off a discussion on Support JSON functions in
> > Flink
> > > > SQL.
> > > > > >
> > > > > > The entire plan is divided into two steps:
> > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink
> SQL[1].
> > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink
> SQL,
> > > > such
> > > > > as
> > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON functions.
> > > > > >
> > > > > > Would love to hear your thoughts.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > > >
> > > > > > Best,
> > > > > > ForwardXu
> > > > >
> > > >
> > >
> >
>

[Jark Wu-2]

Hi Forward,

Sorry for the late reply. I have went through the design doc and I think it
is very nice.

Here are my thoughts and suggestions:

0) I think support JSON functions in SQL is not complicated. Because
Calcite already supports the parser part and the runtime part.
    We only need to integrate it in Flink and add good coverage tests.
1) However, I think we should also design the corresponding JSON Functions
API for Table API which is very important.
    I don't have a clear idea about how to support all the JSON Function
syntax in Table API. And this may need more discussions.
2) IMO, it deserves a FLIP (especially for the Table API part). You can
follow the FLIP process [1] to start a FLIP proposal.
3) I think we only need to implement it in blink planner as we are going to
deprecate old planner.
   So could you update the implementation section in the doc because the
implementation in blink planner should be different.
4) It would be better to have an implementation plan to priority the
sub-tasks.
    From my point of view, JSON_VALUE is the most important and JSON_TABLE
gets the least priority.

I also left some comments in the google doc.

Hi @JingsongLee <[hidden email]> ,

I think we don't need to wait for FLIP-51. As we don't have a clear
progress of FLIP-51.
And as far as I know, it will add a few of PlannerExpressions which can be
refactored easily during FLIP-51.


Cheers,
Jark

[1]:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals



On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]> wrote:

> +1 to have JSON functions in Flink SQL
>
> JingsongLee <[hidden email]> 于2019年9月5日周四 下午4:46写道：
>
> > +1
> > Nice document. I think it is easier to do after expression reworking[1].
> > By the way, which planner do you want to start?
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> >
> > Best,
> > Jingsong Lee
> >
> >
> > ------------------------------------------------------------------
> > From:TANG Wen-hui <[hidden email]>
> > Send Time:2019年9月5日(星期四) 14:36
> > To:dev <[hidden email]>
> > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> >
> > +1
> > I have done similar work before.
> > Looking forward to discussing this feature.
> >
> > Best
> > wenhui
> >
> >
> >
> > [hidden email]
> >
> > From: Kurt Young
> > Date: 2019-09-05 14:00
> > To: dev
> > CC: Anyang Hu
> > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> > +1 to add JSON support to Flink. We also see lots of requirements for
> JSON
> > related functions in our internal platform. Since these are already SQL
> > standard, I think it's a good time to add them to Flink.
> >
> > Best,
> > Kurt
> >
> >
> > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:
> >
> > > We also see strong demands from our SQL users for JSON/Date related
> > > functions.
> > >
> > > Also +Anyang Hu <[hidden email]>
> > >
> > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
> > >
> > > > Hi Forward,
> > > >
> > > > Thanks for bringing this discussion and preparing the nice design.
> > > > I think it's nice to have the JSON functions in the next release.
> > > > We have received some requirements for this feature.
> > > >
> > > > I can help to shepherd this JSON functions effort and will leave
> > comments
> > > >  in the design doc in the next days.
> > > >
> > > > Hi Danny,
> > > >
> > > > The new introduced JSON functions are from SQL:2016, not from MySQL.
> > > > So there no JSON type is needed. According to the SQL:2016, the
> > > > representation of JSON data can be "character string" which is also
> > > > the current implementation in Calcite[1].
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > >
> > > > [1]: https://calcite.apache.org/docs/reference.html#json-functions
> > > >
> > > >
> > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]>
> > wrote:
> > > >
> > > > > hi Danny Chan ,Thank you very much for your reply, your help can
> help
> > > me
> > > > > further improve this discussion.
> > > > > Best
> > > > > forward
> > > > >
> > > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > > > >
> > > > > > Thanks Xu Forward for bring up this topic, I think the JSON
> > functions
> > > > are
> > > > > > very useful especially for those MySQL users.
> > > > > >
> > > > > > I saw that you have done some work within the Apache Calcite,
> > that’s
> > > a
> > > > > > good start, but this is one concern from me, Flink doesn’t
> support
> > > JSON
> > > > > > type internal, so how to represent a JSON object in Flink maybe a
> > key
> > > > > point
> > > > > > we need to resolve. In Calcite, we use ANY type to represent as
> the
> > > > JSON,
> > > > > > but I don’t think it is the right way to go, maybe we can have a
> > > > > discussion
> > > > > > here.
> > > > > >
> > > > > > Best,
> > > > > > Danny Chan
> > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]>，写道：
> > > > > > > Hi everybody,
> > > > > > >
> > > > > > > I'd like to kick off a discussion on Support JSON functions in
> > > Flink
> > > > > SQL.
> > > > > > >
> > > > > > > The entire plan is divided into two steps:
> > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink
> > SQL[1].
> > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in Flink
> > SQL,
> > > > > such
> > > > > > as
> > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON
> functions.
> > > > > > >
> > > > > > > Would love to hear your thoughts.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > > > >
> > > > > > > Best,
> > > > > > > ForwardXu
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Forward Xu]

Hi Jack,
Thank you very much for your reply, google doc I have updated, and some of
your questions I replied.
 In addition, I want to apply for Flip permissions for this purpose.

Best,
Forward

Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：

> Hi Forward,
>
> Sorry for the late reply. I have went through the design doc and I think it
> is very nice.
>
> Here are my thoughts and suggestions:
>
> 0) I think support JSON functions in SQL is not complicated. Because
> Calcite already supports the parser part and the runtime part.
>     We only need to integrate it in Flink and add good coverage tests.
> 1) However, I think we should also design the corresponding JSON Functions
> API for Table API which is very important.
>     I don't have a clear idea about how to support all the JSON Function
> syntax in Table API. And this may need more discussions.
> 2) IMO, it deserves a FLIP (especially for the Table API part). You can
> follow the FLIP process [1] to start a FLIP proposal.
> 3) I think we only need to implement it in blink planner as we are going to
> deprecate old planner.
>    So could you update the implementation section in the doc because the
> implementation in blink planner should be different.
> 4) It would be better to have an implementation plan to priority the
> sub-tasks.
>     From my point of view, JSON_VALUE is the most important and JSON_TABLE
> gets the least priority.
>
> I also left some comments in the google doc.
>
> Hi @JingsongLee <[hidden email]> ,
>
> I think we don't need to wait for FLIP-51. As we don't have a clear
> progress of FLIP-51.
> And as far as I know, it will add a few of PlannerExpressions which can be
> refactored easily during FLIP-51.
>
>
> Cheers,
> Jark
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
>
>
> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]> wrote:
>
> > +1 to have JSON functions in Flink SQL
> >
> > JingsongLee <[hidden email]> 于2019年9月5日周四 下午4:46写道：
> >
> > > +1
> > > Nice document. I think it is easier to do after expression
> reworking[1].
> > > By the way, which planner do you want to start?
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:TANG Wen-hui <[hidden email]>
> > > Send Time:2019年9月5日(星期四) 14:36
> > > To:dev <[hidden email]>
> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> > >
> > > +1
> > > I have done similar work before.
> > > Looking forward to discussing this feature.
> > >
> > > Best
> > > wenhui
> > >
> > >
> > >
> > > [hidden email]
> > >
> > > From: Kurt Young
> > > Date: 2019-09-05 14:00
> > > To: dev
> > > CC: Anyang Hu
> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> > > +1 to add JSON support to Flink. We also see lots of requirements for
> > JSON
> > > related functions in our internal platform. Since these are already SQL
> > > standard, I think it's a good time to add them to Flink.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:
> > >
> > > > We also see strong demands from our SQL users for JSON/Date related
> > > > functions.
> > > >
> > > > Also +Anyang Hu <[hidden email]>
> > > >
> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
> > > >
> > > > > Hi Forward,
> > > > >
> > > > > Thanks for bringing this discussion and preparing the nice design.
> > > > > I think it's nice to have the JSON functions in the next release.
> > > > > We have received some requirements for this feature.
> > > > >
> > > > > I can help to shepherd this JSON functions effort and will leave
> > > comments
> > > > >  in the design doc in the next days.
> > > > >
> > > > > Hi Danny,
> > > > >
> > > > > The new introduced JSON functions are from SQL:2016, not from
> MySQL.
> > > > > So there no JSON type is needed. According to the SQL:2016, the
> > > > > representation of JSON data can be "character string" which is also
> > > > > the current implementation in Calcite[1].
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > >
> > > > > [1]: https://calcite.apache.org/docs/reference.html#json-functions
> > > > >
> > > > >
> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]>
> > > wrote:
> > > > >
> > > > > > hi Danny Chan ,Thank you very much for your reply, your help can
> > help
> > > > me
> > > > > > further improve this discussion.
> > > > > > Best
> > > > > > forward
> > > > > >
> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > > > > >
> > > > > > > Thanks Xu Forward for bring up this topic, I think the JSON
> > > functions
> > > > > are
> > > > > > > very useful especially for those MySQL users.
> > > > > > >
> > > > > > > I saw that you have done some work within the Apache Calcite,
> > > that’s
> > > > a
> > > > > > > good start, but this is one concern from me, Flink doesn’t
> > support
> > > > JSON
> > > > > > > type internal, so how to represent a JSON object in Flink
> maybe a
> > > key
> > > > > > point
> > > > > > > we need to resolve. In Calcite, we use ANY type to represent as
> > the
> > > > > JSON,
> > > > > > > but I don’t think it is the right way to go, maybe we can have
> a
> > > > > > discussion
> > > > > > > here.
> > > > > > >
> > > > > > > Best,
> > > > > > > Danny Chan
> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]
> >，写道：
> > > > > > > > Hi everybody,
> > > > > > > >
> > > > > > > > I'd like to kick off a discussion on Support JSON functions
> in
> > > > Flink
> > > > > > SQL.
> > > > > > > >
> > > > > > > > The entire plan is divided into two steps:
> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink
> > > SQL[1].
> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in
> Flink
> > > SQL,
> > > > > > such
> > > > > > > as
> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON
> > functions.
> > > > > > > >
> > > > > > > > Would love to hear your thoughts.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > ForwardXu
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Forward Xu]

Hi everyone, It's been a long time since I started this discussion. Do you
have anything to add and improve?
Best,
Forward

Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：

> Hi Jack,
> Thank you very much for your reply, google doc I have updated, and some of
> your questions I replied.
>  In addition, I want to apply for Flip permissions for this purpose.
>
> Best,
> Forward
>
> Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
>
>> Hi Forward,
>>
>> Sorry for the late reply. I have went through the design doc and I think
>> it
>> is very nice.
>>
>> Here are my thoughts and suggestions:
>>
>> 0) I think support JSON functions in SQL is not complicated. Because
>> Calcite already supports the parser part and the runtime part.
>>     We only need to integrate it in Flink and add good coverage tests.
>> 1) However, I think we should also design the corresponding JSON Functions
>> API for Table API which is very important.
>>     I don't have a clear idea about how to support all the JSON Function
>> syntax in Table API. And this may need more discussions.
>> 2) IMO, it deserves a FLIP (especially for the Table API part). You can
>> follow the FLIP process [1] to start a FLIP proposal.
>> 3) I think we only need to implement it in blink planner as we are going
>> to
>> deprecate old planner.
>>    So could you update the implementation section in the doc because the
>> implementation in blink planner should be different.
>> 4) It would be better to have an implementation plan to priority the
>> sub-tasks.
>>     From my point of view, JSON_VALUE is the most important and JSON_TABLE
>> gets the least priority.
>>
>> I also left some comments in the google doc.
>>
>> Hi @JingsongLee <[hidden email]> ,
>>
>> I think we don't need to wait for FLIP-51. As we don't have a clear
>> progress of FLIP-51.
>> And as far as I know, it will add a few of PlannerExpressions which can be
>> refactored easily during FLIP-51.
>>
>>
>> Cheers,
>> Jark
>>
>> [1]:
>>
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>>
>>
>>
>> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]> wrote:
>>
>> > +1 to have JSON functions in Flink SQL
>> >
>> > JingsongLee <[hidden email]> 于2019年9月5日周四 下午4:46写道：
>> >
>> > > +1
>> > > Nice document. I think it is easier to do after expression
>> reworking[1].
>> > > By the way, which planner do you want to start?
>> > >
>> > > [1]
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>> > >
>> > > Best,
>> > > Jingsong Lee
>> > >
>> > >
>> > > ------------------------------------------------------------------
>> > > From:TANG Wen-hui <[hidden email]>
>> > > Send Time:2019年9月5日(星期四) 14:36
>> > > To:dev <[hidden email]>
>> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
>> > >
>> > > +1
>> > > I have done similar work before.
>> > > Looking forward to discussing this feature.
>> > >
>> > > Best
>> > > wenhui
>> > >
>> > >
>> > >
>> > > [hidden email]
>> > >
>> > > From: Kurt Young
>> > > Date: 2019-09-05 14:00
>> > > To: dev
>> > > CC: Anyang Hu
>> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
>> > > +1 to add JSON support to Flink. We also see lots of requirements for
>> > JSON
>> > > related functions in our internal platform. Since these are already
>> SQL
>> > > standard, I think it's a good time to add them to Flink.
>> > >
>> > > Best,
>> > > Kurt
>> > >
>> > >
>> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:
>> > >
>> > > > We also see strong demands from our SQL users for JSON/Date related
>> > > > functions.
>> > > >
>> > > > Also +Anyang Hu <[hidden email]>
>> > > >
>> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
>> > > >
>> > > > > Hi Forward,
>> > > > >
>> > > > > Thanks for bringing this discussion and preparing the nice design.
>> > > > > I think it's nice to have the JSON functions in the next release.
>> > > > > We have received some requirements for this feature.
>> > > > >
>> > > > > I can help to shepherd this JSON functions effort and will leave
>> > > comments
>> > > > >  in the design doc in the next days.
>> > > > >
>> > > > > Hi Danny,
>> > > > >
>> > > > > The new introduced JSON functions are from SQL:2016, not from
>> MySQL.
>> > > > > So there no JSON type is needed. According to the SQL:2016, the
>> > > > > representation of JSON data can be "character string" which is
>> also
>> > > > > the current implementation in Calcite[1].
>> > > > >
>> > > > > Best,
>> > > > > Jark
>> > > > >
>> > > > >
>> > > > > [1]:
>> https://calcite.apache.org/docs/reference.html#json-functions
>> > > > >
>> > > > >
>> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]>
>> > > wrote:
>> > > > >
>> > > > > > hi Danny Chan ,Thank you very much for your reply, your help can
>> > help
>> > > > me
>> > > > > > further improve this discussion.
>> > > > > > Best
>> > > > > > forward
>> > > > > >
>> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
>> > > > > >
>> > > > > > > Thanks Xu Forward for bring up this topic, I think the JSON
>> > > functions
>> > > > > are
>> > > > > > > very useful especially for those MySQL users.
>> > > > > > >
>> > > > > > > I saw that you have done some work within the Apache Calcite,
>> > > that’s
>> > > > a
>> > > > > > > good start, but this is one concern from me, Flink doesn’t
>> > support
>> > > > JSON
>> > > > > > > type internal, so how to represent a JSON object in Flink
>> maybe a
>> > > key
>> > > > > > point
>> > > > > > > we need to resolve. In Calcite, we use ANY type to represent
>> as
>> > the
>> > > > > JSON,
>> > > > > > > but I don’t think it is the right way to go, maybe we can
>> have a
>> > > > > > discussion
>> > > > > > > here.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Danny Chan
>> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]
>> >，写道：
>> > > > > > > > Hi everybody,
>> > > > > > > >
>> > > > > > > > I'd like to kick off a discussion on Support JSON functions
>> in
>> > > > Flink
>> > > > > > SQL.
>> > > > > > > >
>> > > > > > > > The entire plan is divided into two steps:
>> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink
>> > > SQL[1].
>> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in
>> Flink
>> > > SQL,
>> > > > > > such
>> > > > > > > as
>> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON
>> > functions.
>> > > > > > > >
>> > > > > > > > Would love to hear your thoughts.
>> > > > > > > >
>> > > > > > > > [1]
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > ForwardXu
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

[Jingsong Li]

Hi Forward:

Document looks good to me.
I think you can just start doing this.
They all work very independently, so I don't think there's any obvious
blocking.

Best,
Jingsong Lee

On Sat, Nov 30, 2019 at 10:59 AM Forward Xu <[hidden email]> wrote:

> Hi everyone, It's been a long time since I started this discussion. Do you
> have anything to add and improve?
> Best,
> Forward
>
> Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：
>
> > Hi Jack,
> > Thank you very much for your reply, google doc I have updated, and some
> of
> > your questions I replied.
> >  In addition, I want to apply for Flip permissions for this purpose.
> >
> > Best,
> > Forward
> >
> > Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
> >
> >> Hi Forward,
> >>
> >> Sorry for the late reply. I have went through the design doc and I think
> >> it
> >> is very nice.
> >>
> >> Here are my thoughts and suggestions:
> >>
> >> 0) I think support JSON functions in SQL is not complicated. Because
> >> Calcite already supports the parser part and the runtime part.
> >>     We only need to integrate it in Flink and add good coverage tests.
> >> 1) However, I think we should also design the corresponding JSON
> Functions
> >> API for Table API which is very important.
> >>     I don't have a clear idea about how to support all the JSON Function
> >> syntax in Table API. And this may need more discussions.
> >> 2) IMO, it deserves a FLIP (especially for the Table API part). You can
> >> follow the FLIP process [1] to start a FLIP proposal.
> >> 3) I think we only need to implement it in blink planner as we are going
> >> to
> >> deprecate old planner.
> >>    So could you update the implementation section in the doc because the
> >> implementation in blink planner should be different.
> >> 4) It would be better to have an implementation plan to priority the
> >> sub-tasks.
> >>     From my point of view, JSON_VALUE is the most important and
> JSON_TABLE
> >> gets the least priority.
> >>
> >> I also left some comments in the google doc.
> >>
> >> Hi @JingsongLee <[hidden email]> ,
> >>
> >> I think we don't need to wait for FLIP-51. As we don't have a clear
> >> progress of FLIP-51.
> >> And as far as I know, it will add a few of PlannerExpressions which can
> be
> >> refactored easily during FLIP-51.
> >>
> >>
> >> Cheers,
> >> Jark
> >>
> >> [1]:
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >>
> >>
> >>
> >> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]> wrote:
> >>
> >> > +1 to have JSON functions in Flink SQL
> >> >
> >> > JingsongLee <[hidden email]> 于2019年9月5日周四 下午4:46写道：
> >> >
> >> > > +1
> >> > > Nice document. I think it is easier to do after expression
> >> reworking[1].
> >> > > By the way, which planner do you want to start?
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> >> > >
> >> > > Best,
> >> > > Jingsong Lee
> >> > >
> >> > >
> >> > > ------------------------------------------------------------------
> >> > > From:TANG Wen-hui <[hidden email]>
> >> > > Send Time:2019年9月5日(星期四) 14:36
> >> > > To:dev <[hidden email]>
> >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> >> > >
> >> > > +1
> >> > > I have done similar work before.
> >> > > Looking forward to discussing this feature.
> >> > >
> >> > > Best
> >> > > wenhui
> >> > >
> >> > >
> >> > >
> >> > > [hidden email]
> >> > >
> >> > > From: Kurt Young
> >> > > Date: 2019-09-05 14:00
> >> > > To: dev
> >> > > CC: Anyang Hu
> >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> >> > > +1 to add JSON support to Flink. We also see lots of requirements
> for
> >> > JSON
> >> > > related functions in our internal platform. Since these are already
> >> SQL
> >> > > standard, I think it's a good time to add them to Flink.
> >> > >
> >> > > Best,
> >> > > Kurt
> >> > >
> >> > >
> >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]> wrote:
> >> > >
> >> > > > We also see strong demands from our SQL users for JSON/Date
> related
> >> > > > functions.
> >> > > >
> >> > > > Also +Anyang Hu <[hidden email]>
> >> > > >
> >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]> wrote:
> >> > > >
> >> > > > > Hi Forward,
> >> > > > >
> >> > > > > Thanks for bringing this discussion and preparing the nice
> design.
> >> > > > > I think it's nice to have the JSON functions in the next
> release.
> >> > > > > We have received some requirements for this feature.
> >> > > > >
> >> > > > > I can help to shepherd this JSON functions effort and will leave
> >> > > comments
> >> > > > >  in the design doc in the next days.
> >> > > > >
> >> > > > > Hi Danny,
> >> > > > >
> >> > > > > The new introduced JSON functions are from SQL:2016, not from
> >> MySQL.
> >> > > > > So there no JSON type is needed. According to the SQL:2016, the
> >> > > > > representation of JSON data can be "character string" which is
> >> also
> >> > > > > the current implementation in Calcite[1].
> >> > > > >
> >> > > > > Best,
> >> > > > > Jark
> >> > > > >
> >> > > > >
> >> > > > > [1]:
> >> https://calcite.apache.org/docs/reference.html#json-functions
> >> > > > >
> >> > > > >
> >> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <[hidden email]
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > hi Danny Chan ,Thank you very much for your reply, your help
> can
> >> > help
> >> > > > me
> >> > > > > > further improve this discussion.
> >> > > > > > Best
> >> > > > > > forward
> >> > > > > >
> >> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> >> > > > > >
> >> > > > > > > Thanks Xu Forward for bring up this topic, I think the JSON
> >> > > functions
> >> > > > > are
> >> > > > > > > very useful especially for those MySQL users.
> >> > > > > > >
> >> > > > > > > I saw that you have done some work within the Apache
> Calcite,
> >> > > that’s
> >> > > > a
> >> > > > > > > good start, but this is one concern from me, Flink doesn’t
> >> > support
> >> > > > JSON
> >> > > > > > > type internal, so how to represent a JSON object in Flink
> >> maybe a
> >> > > key
> >> > > > > > point
> >> > > > > > > we need to resolve. In Calcite, we use ANY type to represent
> >> as
> >> > the
> >> > > > > JSON,
> >> > > > > > > but I don’t think it is the right way to go, maybe we can
> >> have a
> >> > > > > > discussion
> >> > > > > > > here.
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Danny Chan
> >> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <[hidden email]
> >> >，写道：
> >> > > > > > > > Hi everybody,
> >> > > > > > > >
> >> > > > > > > > I'd like to kick off a discussion on Support JSON
> functions
> >> in
> >> > > > Flink
> >> > > > > > SQL.
> >> > > > > > > >
> >> > > > > > > > The entire plan is divided into two steps:
> >> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in Flink
> >> > > SQL[1].
> >> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in
> >> Flink
> >> > > SQL,
> >> > > > > > such
> >> > > > > > > as
> >> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON
> >> > functions.
> >> > > > > > > >
> >> > > > > > > > Would love to hear your thoughts.
> >> > > > > > > >
> >> > > > > > > > [1]
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> >> > > > > > > >
> >> > > > > > > > Best,
> >> > > > > > > > ForwardXu
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

--
Best, Jingsong Lee

[Forward Xu]

Hi Jingsong Lee :

Thank you very much, I need to apply for FLIP permission. Do I need to
create a FLIP for this?

Best,
Forward

Jingsong Li <[hidden email]> 于2019年12月2日周一 上午11:40写道：

> Hi Forward:
>
> Document looks good to me.
> I think you can just start doing this.
> They all work very independently, so I don't think there's any obvious
> blocking.
>
> Best,
> Jingsong Lee
>
> On Sat, Nov 30, 2019 at 10:59 AM Forward Xu <[hidden email]>
> wrote:
>
> > Hi everyone, It's been a long time since I started this discussion. Do
> you
> > have anything to add and improve?
> > Best,
> > Forward
> >
> > Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：
> >
> > > Hi Jack,
> > > Thank you very much for your reply, google doc I have updated, and some
> > of
> > > your questions I replied.
> > >  In addition, I want to apply for Flip permissions for this purpose.
> > >
> > > Best,
> > > Forward
> > >
> > > Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
> > >
> > >> Hi Forward,
> > >>
> > >> Sorry for the late reply. I have went through the design doc and I
> think
> > >> it
> > >> is very nice.
> > >>
> > >> Here are my thoughts and suggestions:
> > >>
> > >> 0) I think support JSON functions in SQL is not complicated. Because
> > >> Calcite already supports the parser part and the runtime part.
> > >>     We only need to integrate it in Flink and add good coverage tests.
> > >> 1) However, I think we should also design the corresponding JSON
> > Functions
> > >> API for Table API which is very important.
> > >>     I don't have a clear idea about how to support all the JSON
> Function
> > >> syntax in Table API. And this may need more discussions.
> > >> 2) IMO, it deserves a FLIP (especially for the Table API part). You
> can
> > >> follow the FLIP process [1] to start a FLIP proposal.
> > >> 3) I think we only need to implement it in blink planner as we are
> going
> > >> to
> > >> deprecate old planner.
> > >>    So could you update the implementation section in the doc because
> the
> > >> implementation in blink planner should be different.
> > >> 4) It would be better to have an implementation plan to priority the
> > >> sub-tasks.
> > >>     From my point of view, JSON_VALUE is the most important and
> > JSON_TABLE
> > >> gets the least priority.
> > >>
> > >> I also left some comments in the google doc.
> > >>
> > >> Hi @JingsongLee <[hidden email]> ,
> > >>
> > >> I think we don't need to wait for FLIP-51. As we don't have a clear
> > >> progress of FLIP-51.
> > >> And as far as I know, it will add a few of PlannerExpressions which
> can
> > be
> > >> refactored easily during FLIP-51.
> > >>
> > >>
> > >> Cheers,
> > >> Jark
> > >>
> > >> [1]:
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >>
> > >>
> > >>
> > >> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]> wrote:
> > >>
> > >> > +1 to have JSON functions in Flink SQL
> > >> >
> > >> > JingsongLee <[hidden email]> 于2019年9月5日周四
> 下午4:46写道：
> > >> >
> > >> > > +1
> > >> > > Nice document. I think it is easier to do after expression
> > >> reworking[1].
> > >> > > By the way, which planner do you want to start?
> > >> > >
> > >> > > [1]
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > >> > >
> > >> > > Best,
> > >> > > Jingsong Lee
> > >> > >
> > >> > >
> > >> > > ------------------------------------------------------------------
> > >> > > From:TANG Wen-hui <[hidden email]>
> > >> > > Send Time:2019年9月5日(星期四) 14:36
> > >> > > To:dev <[hidden email]>
> > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> > >> > >
> > >> > > +1
> > >> > > I have done similar work before.
> > >> > > Looking forward to discussing this feature.
> > >> > >
> > >> > > Best
> > >> > > wenhui
> > >> > >
> > >> > >
> > >> > >
> > >> > > [hidden email]
> > >> > >
> > >> > > From: Kurt Young
> > >> > > Date: 2019-09-05 14:00
> > >> > > To: dev
> > >> > > CC: Anyang Hu
> > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> > >> > > +1 to add JSON support to Flink. We also see lots of requirements
> > for
> > >> > JSON
> > >> > > related functions in our internal platform. Since these are
> already
> > >> SQL
> > >> > > standard, I think it's a good time to add them to Flink.
> > >> > >
> > >> > > Best,
> > >> > > Kurt
> > >> > >
> > >> > >
> > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]>
> wrote:
> > >> > >
> > >> > > > We also see strong demands from our SQL users for JSON/Date
> > related
> > >> > > > functions.
> > >> > > >
> > >> > > > Also +Anyang Hu <[hidden email]>
> > >> > > >
> > >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]>
> wrote:
> > >> > > >
> > >> > > > > Hi Forward,
> > >> > > > >
> > >> > > > > Thanks for bringing this discussion and preparing the nice
> > design.
> > >> > > > > I think it's nice to have the JSON functions in the next
> > release.
> > >> > > > > We have received some requirements for this feature.
> > >> > > > >
> > >> > > > > I can help to shepherd this JSON functions effort and will
> leave
> > >> > > comments
> > >> > > > >  in the design doc in the next days.
> > >> > > > >
> > >> > > > > Hi Danny,
> > >> > > > >
> > >> > > > > The new introduced JSON functions are from SQL:2016, not from
> > >> MySQL.
> > >> > > > > So there no JSON type is needed. According to the SQL:2016,
> the
> > >> > > > > representation of JSON data can be "character string" which is
> > >> also
> > >> > > > > the current implementation in Calcite[1].
> > >> > > > >
> > >> > > > > Best,
> > >> > > > > Jark
> > >> > > > >
> > >> > > > >
> > >> > > > > [1]:
> > >> https://calcite.apache.org/docs/reference.html#json-functions
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <
> [hidden email]
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > hi Danny Chan ,Thank you very much for your reply, your help
> > can
> > >> > help
> > >> > > > me
> > >> > > > > > further improve this discussion.
> > >> > > > > > Best
> > >> > > > > > forward
> > >> > > > > >
> > >> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > >> > > > > >
> > >> > > > > > > Thanks Xu Forward for bring up this topic, I think the
> JSON
> > >> > > functions
> > >> > > > > are
> > >> > > > > > > very useful especially for those MySQL users.
> > >> > > > > > >
> > >> > > > > > > I saw that you have done some work within the Apache
> > Calcite,
> > >> > > that’s
> > >> > > > a
> > >> > > > > > > good start, but this is one concern from me, Flink doesn’t
> > >> > support
> > >> > > > JSON
> > >> > > > > > > type internal, so how to represent a JSON object in Flink
> > >> maybe a
> > >> > > key
> > >> > > > > > point
> > >> > > > > > > we need to resolve. In Calcite, we use ANY type to
> represent
> > >> as
> > >> > the
> > >> > > > > JSON,
> > >> > > > > > > but I don’t think it is the right way to go, maybe we can
> > >> have a
> > >> > > > > > discussion
> > >> > > > > > > here.
> > >> > > > > > >
> > >> > > > > > > Best,
> > >> > > > > > > Danny Chan
> > >> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <
> [hidden email]
> > >> >，写道：
> > >> > > > > > > > Hi everybody,
> > >> > > > > > > >
> > >> > > > > > > > I'd like to kick off a discussion on Support JSON
> > functions
> > >> in
> > >> > > > Flink
> > >> > > > > > SQL.
> > >> > > > > > > >
> > >> > > > > > > > The entire plan is divided into two steps:
> > >> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in
> Flink
> > >> > > SQL[1].
> > >> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions in
> > >> Flink
> > >> > > SQL,
> > >> > > > > > such
> > >> > > > > > > as
> > >> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON
> > >> > functions.
> > >> > > > > > > >
> > >> > > > > > > > Would love to hear your thoughts.
> > >> > > > > > > >
> > >> > > > > > > > [1]
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > >> > > > > > > >
> > >> > > > > > > > Best,
> > >> > > > > > > > ForwardXu
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

[Jark Wu-2]

Hi Forward,

Sorry for the late reply.
As I said before, it would be better to include the JSON functions API for
Table API.
For example, what the "json_exist", "json_value", "json_query", etc...
functions in Table API looks like.
And it would be better to follow FLIP template [1] which include "Public
Interface" (brief list of public interface), and "Proposed changes"
(detailed proposal).

Once the design doc looks good, you can update it to FLIP and start a vote.

Btw, what's your username in confluence?


Best,
Jark


[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template


On Mon, 2 Dec 2019 at 11:44, Forward Xu <[hidden email]> wrote:

> Hi Jingsong Lee :
>
> Thank you very much, I need to apply for FLIP permission. Do I need to
> create a FLIP for this?
>
> Best,
> Forward
>
> Jingsong Li <[hidden email]> 于2019年12月2日周一 上午11:40写道：
>
> > Hi Forward:
> >
> > Document looks good to me.
> > I think you can just start doing this.
> > They all work very independently, so I don't think there's any obvious
> > blocking.
> >
> > Best,
> > Jingsong Lee
> >
> > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu <[hidden email]>
> > wrote:
> >
> > > Hi everyone, It's been a long time since I started this discussion. Do
> > you
> > > have anything to add and improve?
> > > Best,
> > > Forward
> > >
> > > Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：
> > >
> > > > Hi Jack,
> > > > Thank you very much for your reply, google doc I have updated, and
> some
> > > of
> > > > your questions I replied.
> > > >  In addition, I want to apply for Flip permissions for this purpose.
> > > >
> > > > Best,
> > > > Forward
> > > >
> > > > Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
> > > >
> > > >> Hi Forward,
> > > >>
> > > >> Sorry for the late reply. I have went through the design doc and I
> > think
> > > >> it
> > > >> is very nice.
> > > >>
> > > >> Here are my thoughts and suggestions:
> > > >>
> > > >> 0) I think support JSON functions in SQL is not complicated. Because
> > > >> Calcite already supports the parser part and the runtime part.
> > > >>     We only need to integrate it in Flink and add good coverage
> tests.
> > > >> 1) However, I think we should also design the corresponding JSON
> > > Functions
> > > >> API for Table API which is very important.
> > > >>     I don't have a clear idea about how to support all the JSON
> > Function
> > > >> syntax in Table API. And this may need more discussions.
> > > >> 2) IMO, it deserves a FLIP (especially for the Table API part). You
> > can
> > > >> follow the FLIP process [1] to start a FLIP proposal.
> > > >> 3) I think we only need to implement it in blink planner as we are
> > going
> > > >> to
> > > >> deprecate old planner.
> > > >>    So could you update the implementation section in the doc because
> > the
> > > >> implementation in blink planner should be different.
> > > >> 4) It would be better to have an implementation plan to priority the
> > > >> sub-tasks.
> > > >>     From my point of view, JSON_VALUE is the most important and
> > > JSON_TABLE
> > > >> gets the least priority.
> > > >>
> > > >> I also left some comments in the google doc.
> > > >>
> > > >> Hi @JingsongLee <[hidden email]> ,
> > > >>
> > > >> I think we don't need to wait for FLIP-51. As we don't have a clear
> > > >> progress of FLIP-51.
> > > >> And as far as I know, it will add a few of PlannerExpressions which
> > can
> > > be
> > > >> refactored easily during FLIP-51.
> > > >>
> > > >>
> > > >> Cheers,
> > > >> Jark
> > > >>
> > > >> [1]:
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >>
> > > >>
> > > >>
> > > >> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]>
> wrote:
> > > >>
> > > >> > +1 to have JSON functions in Flink SQL
> > > >> >
> > > >> > JingsongLee <[hidden email]> 于2019年9月5日周四
> > 下午4:46写道：
> > > >> >
> > > >> > > +1
> > > >> > > Nice document. I think it is easier to do after expression
> > > >> reworking[1].
> > > >> > > By the way, which planner do you want to start?
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > > >> > >
> > > >> > > Best,
> > > >> > > Jingsong Lee
> > > >> > >
> > > >> > >
> > > >> > >
> ------------------------------------------------------------------
> > > >> > > From:TANG Wen-hui <[hidden email]>
> > > >> > > Send Time:2019年9月5日(星期四) 14:36
> > > >> > > To:dev <[hidden email]>
> > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> > > >> > >
> > > >> > > +1
> > > >> > > I have done similar work before.
> > > >> > > Looking forward to discussing this feature.
> > > >> > >
> > > >> > > Best
> > > >> > > wenhui
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > [hidden email]
> > > >> > >
> > > >> > > From: Kurt Young
> > > >> > > Date: 2019-09-05 14:00
> > > >> > > To: dev
> > > >> > > CC: Anyang Hu
> > > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> > > >> > > +1 to add JSON support to Flink. We also see lots of
> requirements
> > > for
> > > >> > JSON
> > > >> > > related functions in our internal platform. Since these are
> > already
> > > >> SQL
> > > >> > > standard, I think it's a good time to add them to Flink.
> > > >> > >
> > > >> > > Best,
> > > >> > > Kurt
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]>
> > wrote:
> > > >> > >
> > > >> > > > We also see strong demands from our SQL users for JSON/Date
> > > related
> > > >> > > > functions.
> > > >> > > >
> > > >> > > > Also +Anyang Hu <[hidden email]>
> > > >> > > >
> > > >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]>
> > wrote:
> > > >> > > >
> > > >> > > > > Hi Forward,
> > > >> > > > >
> > > >> > > > > Thanks for bringing this discussion and preparing the nice
> > > design.
> > > >> > > > > I think it's nice to have the JSON functions in the next
> > > release.
> > > >> > > > > We have received some requirements for this feature.
> > > >> > > > >
> > > >> > > > > I can help to shepherd this JSON functions effort and will
> > leave
> > > >> > > comments
> > > >> > > > >  in the design doc in the next days.
> > > >> > > > >
> > > >> > > > > Hi Danny,
> > > >> > > > >
> > > >> > > > > The new introduced JSON functions are from SQL:2016, not
> from
> > > >> MySQL.
> > > >> > > > > So there no JSON type is needed. According to the SQL:2016,
> > the
> > > >> > > > > representation of JSON data can be "character string" which
> is
> > > >> also
> > > >> > > > > the current implementation in Calcite[1].
> > > >> > > > >
> > > >> > > > > Best,
> > > >> > > > > Jark
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > [1]:
> > > >> https://calcite.apache.org/docs/reference.html#json-functions
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <
> > [hidden email]
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > hi Danny Chan ,Thank you very much for your reply, your
> help
> > > can
> > > >> > help
> > > >> > > > me
> > > >> > > > > > further improve this discussion.
> > > >> > > > > > Best
> > > >> > > > > > forward
> > > >> > > > > >
> > > >> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三 下午8:50写道：
> > > >> > > > > >
> > > >> > > > > > > Thanks Xu Forward for bring up this topic, I think the
> > JSON
> > > >> > > functions
> > > >> > > > > are
> > > >> > > > > > > very useful especially for those MySQL users.
> > > >> > > > > > >
> > > >> > > > > > > I saw that you have done some work within the Apache
> > > Calcite,
> > > >> > > that’s
> > > >> > > > a
> > > >> > > > > > > good start, but this is one concern from me, Flink
> doesn’t
> > > >> > support
> > > >> > > > JSON
> > > >> > > > > > > type internal, so how to represent a JSON object in
> Flink
> > > >> maybe a
> > > >> > > key
> > > >> > > > > > point
> > > >> > > > > > > we need to resolve. In Calcite, we use ANY type to
> > represent
> > > >> as
> > > >> > the
> > > >> > > > > JSON,
> > > >> > > > > > > but I don’t think it is the right way to go, maybe we
> can
> > > >> have a
> > > >> > > > > > discussion
> > > >> > > > > > > here.
> > > >> > > > > > >
> > > >> > > > > > > Best,
> > > >> > > > > > > Danny Chan
> > > >> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <
> > [hidden email]
> > > >> >，写道：
> > > >> > > > > > > > Hi everybody,
> > > >> > > > > > > >
> > > >> > > > > > > > I'd like to kick off a discussion on Support JSON
> > > functions
> > > >> in
> > > >> > > > Flink
> > > >> > > > > > SQL.
> > > >> > > > > > > >
> > > >> > > > > > > > The entire plan is divided into two steps:
> > > >> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in
> > Flink
> > > >> > > SQL[1].
> > > >> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON functions
> in
> > > >> Flink
> > > >> > > SQL,
> > > >> > > > > > such
> > > >> > > > > > > as
> > > >> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful JSON
> > > >> > functions.
> > > >> > > > > > > >
> > > >> > > > > > > > Would love to hear your thoughts.
> > > >> > > > > > > >
> > > >> > > > > > > > [1]
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > >> > > > > > > >
> > > >> > > > > > > > Best,
> > > >> > > > > > > > ForwardXu
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

[Forward Xu]

Hi Jark,
Thank you very much，I will improve this document as soon as possible.
My confluence username is ForwardXu.

Best,
Forward

Jark Wu <[hidden email]> 于2019年12月2日周一 上午11:56写道：

> Hi Forward,
>
> Sorry for the late reply.
> As I said before, it would be better to include the JSON functions API for
> Table API.
> For example, what the "json_exist", "json_value", "json_query", etc...
> functions in Table API looks like.
> And it would be better to follow FLIP template [1] which include "Public
> Interface" (brief list of public interface), and "Proposed changes"
> (detailed proposal).
>
> Once the design doc looks good, you can update it to FLIP and start a vote.
>
> Btw, what's your username in confluence?
>
>
> Best,
> Jark
>
>
> [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template
>
>
> On Mon, 2 Dec 2019 at 11:44, Forward Xu <[hidden email]> wrote:
>
> > Hi Jingsong Lee :
> >
> > Thank you very much, I need to apply for FLIP permission. Do I need to
> > create a FLIP for this?
> >
> > Best,
> > Forward
> >
> > Jingsong Li <[hidden email]> 于2019年12月2日周一 上午11:40写道：
> >
> > > Hi Forward:
> > >
> > > Document looks good to me.
> > > I think you can just start doing this.
> > > They all work very independently, so I don't think there's any obvious
> > > blocking.
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu <[hidden email]>
> > > wrote:
> > >
> > > > Hi everyone, It's been a long time since I started this discussion.
> Do
> > > you
> > > > have anything to add and improve?
> > > > Best,
> > > > Forward
> > > >
> > > > Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：
> > > >
> > > > > Hi Jack,
> > > > > Thank you very much for your reply, google doc I have updated, and
> > some
> > > > of
> > > > > your questions I replied.
> > > > >  In addition, I want to apply for Flip permissions for this
> purpose.
> > > > >
> > > > > Best,
> > > > > Forward
> > > > >
> > > > > Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
> > > > >
> > > > >> Hi Forward,
> > > > >>
> > > > >> Sorry for the late reply. I have went through the design doc and I
> > > think
> > > > >> it
> > > > >> is very nice.
> > > > >>
> > > > >> Here are my thoughts and suggestions:
> > > > >>
> > > > >> 0) I think support JSON functions in SQL is not complicated.
> Because
> > > > >> Calcite already supports the parser part and the runtime part.
> > > > >>     We only need to integrate it in Flink and add good coverage
> > tests.
> > > > >> 1) However, I think we should also design the corresponding JSON
> > > > Functions
> > > > >> API for Table API which is very important.
> > > > >>     I don't have a clear idea about how to support all the JSON
> > > Function
> > > > >> syntax in Table API. And this may need more discussions.
> > > > >> 2) IMO, it deserves a FLIP (especially for the Table API part).
> You
> > > can
> > > > >> follow the FLIP process [1] to start a FLIP proposal.
> > > > >> 3) I think we only need to implement it in blink planner as we are
> > > going
> > > > >> to
> > > > >> deprecate old planner.
> > > > >>    So could you update the implementation section in the doc
> because
> > > the
> > > > >> implementation in blink planner should be different.
> > > > >> 4) It would be better to have an implementation plan to priority
> the
> > > > >> sub-tasks.
> > > > >>     From my point of view, JSON_VALUE is the most important and
> > > > JSON_TABLE
> > > > >> gets the least priority.
> > > > >>
> > > > >> I also left some comments in the google doc.
> > > > >>
> > > > >> Hi @JingsongLee <[hidden email]> ,
> > > > >>
> > > > >> I think we don't need to wait for FLIP-51. As we don't have a
> clear
> > > > >> progress of FLIP-51.
> > > > >> And as far as I know, it will add a few of PlannerExpressions
> which
> > > can
> > > > be
> > > > >> refactored easily during FLIP-51.
> > > > >>
> > > > >>
> > > > >> Cheers,
> > > > >> Jark
> > > > >>
> > > > >> [1]:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]>
> > wrote:
> > > > >>
> > > > >> > +1 to have JSON functions in Flink SQL
> > > > >> >
> > > > >> > JingsongLee <[hidden email]> 于2019年9月5日周四
> > > 下午4:46写道：
> > > > >> >
> > > > >> > > +1
> > > > >> > > Nice document. I think it is easier to do after expression
> > > > >> reworking[1].
> > > > >> > > By the way, which planner do you want to start?
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Jingsong Lee
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > ------------------------------------------------------------------
> > > > >> > > From:TANG Wen-hui <[hidden email]>
> > > > >> > > Send Time:2019年9月5日(星期四) 14:36
> > > > >> > > To:dev <[hidden email]>
> > > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink SQL
> > > > >> > >
> > > > >> > > +1
> > > > >> > > I have done similar work before.
> > > > >> > > Looking forward to discussing this feature.
> > > > >> > >
> > > > >> > > Best
> > > > >> > > wenhui
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > [hidden email]
> > > > >> > >
> > > > >> > > From: Kurt Young
> > > > >> > > Date: 2019-09-05 14:00
> > > > >> > > To: dev
> > > > >> > > CC: Anyang Hu
> > > > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> > > > >> > > +1 to add JSON support to Flink. We also see lots of
> > requirements
> > > > for
> > > > >> > JSON
> > > > >> > > related functions in our internal platform. Since these are
> > > already
> > > > >> SQL
> > > > >> > > standard, I think it's a good time to add them to Flink.
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Kurt
> > > > >> > >
> > > > >> > >
> > > > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]>
> > > wrote:
> > > > >> > >
> > > > >> > > > We also see strong demands from our SQL users for JSON/Date
> > > > related
> > > > >> > > > functions.
> > > > >> > > >
> > > > >> > > > Also +Anyang Hu <[hidden email]>
> > > > >> > > >
> > > > >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]>
> > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Forward,
> > > > >> > > > >
> > > > >> > > > > Thanks for bringing this discussion and preparing the nice
> > > > design.
> > > > >> > > > > I think it's nice to have the JSON functions in the next
> > > > release.
> > > > >> > > > > We have received some requirements for this feature.
> > > > >> > > > >
> > > > >> > > > > I can help to shepherd this JSON functions effort and will
> > > leave
> > > > >> > > comments
> > > > >> > > > >  in the design doc in the next days.
> > > > >> > > > >
> > > > >> > > > > Hi Danny,
> > > > >> > > > >
> > > > >> > > > > The new introduced JSON functions are from SQL:2016, not
> > from
> > > > >> MySQL.
> > > > >> > > > > So there no JSON type is needed. According to the
> SQL:2016,
> > > the
> > > > >> > > > > representation of JSON data can be "character string"
> which
> > is
> > > > >> also
> > > > >> > > > > the current implementation in Calcite[1].
> > > > >> > > > >
> > > > >> > > > > Best,
> > > > >> > > > > Jark
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > [1]:
> > > > >> https://calcite.apache.org/docs/reference.html#json-functions
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <
> > > [hidden email]
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > hi Danny Chan ,Thank you very much for your reply, your
> > help
> > > > can
> > > > >> > help
> > > > >> > > > me
> > > > >> > > > > > further improve this discussion.
> > > > >> > > > > > Best
> > > > >> > > > > > forward
> > > > >> > > > > >
> > > > >> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三
> 下午8:50写道：
> > > > >> > > > > >
> > > > >> > > > > > > Thanks Xu Forward for bring up this topic, I think the
> > > JSON
> > > > >> > > functions
> > > > >> > > > > are
> > > > >> > > > > > > very useful especially for those MySQL users.
> > > > >> > > > > > >
> > > > >> > > > > > > I saw that you have done some work within the Apache
> > > > Calcite,
> > > > >> > > that’s
> > > > >> > > > a
> > > > >> > > > > > > good start, but this is one concern from me, Flink
> > doesn’t
> > > > >> > support
> > > > >> > > > JSON
> > > > >> > > > > > > type internal, so how to represent a JSON object in
> > Flink
> > > > >> maybe a
> > > > >> > > key
> > > > >> > > > > > point
> > > > >> > > > > > > we need to resolve. In Calcite, we use ANY type to
> > > represent
> > > > >> as
> > > > >> > the
> > > > >> > > > > JSON,
> > > > >> > > > > > > but I don’t think it is the right way to go, maybe we
> > can
> > > > >> have a
> > > > >> > > > > > discussion
> > > > >> > > > > > > here.
> > > > >> > > > > > >
> > > > >> > > > > > > Best,
> > > > >> > > > > > > Danny Chan
> > > > >> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <
> > > [hidden email]
> > > > >> >，写道：
> > > > >> > > > > > > > Hi everybody,
> > > > >> > > > > > > >
> > > > >> > > > > > > > I'd like to kick off a discussion on Support JSON
> > > > functions
> > > > >> in
> > > > >> > > > Flink
> > > > >> > > > > > SQL.
> > > > >> > > > > > > >
> > > > >> > > > > > > > The entire plan is divided into two steps:
> > > > >> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions in
> > > Flink
> > > > >> > > SQL[1].
> > > > >> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON
> functions
> > in
> > > > >> Flink
> > > > >> > > SQL,
> > > > >> > > > > > such
> > > > >> > > > > > > as
> > > > >> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful
> JSON
> > > > >> > functions.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Would love to hear your thoughts.
> > > > >> > > > > > > >
> > > > >> > > > > > > > [1]
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > >> > > > > > > >
> > > > >> > > > > > > > Best,
> > > > >> > > > > > > > ForwardXu
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best, Jingsong Lee
> > >
> >
>

[Jark Wu-2]

Hi Forward,

Thanks for updating the documentation. I think it is

On Mon, 2 Dec 2019 at 14:21, Forward Xu <[hidden email]> wrote:

> Hi Jark,
> Thank you very much，I will improve this document as soon as possible.
> My confluence username is ForwardXu.
>
> Best,
> Forward
>
> Jark Wu <[hidden email]> 于2019年12月2日周一 上午11:56写道：
>
> > Hi Forward,
> >
> > Sorry for the late reply.
> > As I said before, it would be better to include the JSON functions API
> for
> > Table API.
> > For example, what the "json_exist", "json_value", "json_query", etc...
> > functions in Table API looks like.
> > And it would be better to follow FLIP template [1] which include "Public
> > Interface" (brief list of public interface), and "Proposed changes"
> > (detailed proposal).
> >
> > Once the design doc looks good, you can update it to FLIP and start a
> vote.
> >
> > Btw, what's your username in confluence?
> >
> >
> > Best,
> > Jark
> >
> >
> > [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template
> >
> >
> > On Mon, 2 Dec 2019 at 11:44, Forward Xu <[hidden email]> wrote:
> >
> > > Hi Jingsong Lee :
> > >
> > > Thank you very much, I need to apply for FLIP permission. Do I need to
> > > create a FLIP for this?
> > >
> > > Best,
> > > Forward
> > >
> > > Jingsong Li <[hidden email]> 于2019年12月2日周一 上午11:40写道：
> > >
> > > > Hi Forward:
> > > >
> > > > Document looks good to me.
> > > > I think you can just start doing this.
> > > > They all work very independently, so I don't think there's any
> obvious
> > > > blocking.
> > > >
> > > > Best,
> > > > Jingsong Lee
> > > >
> > > > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu <[hidden email]>
> > > > wrote:
> > > >
> > > > > Hi everyone, It's been a long time since I started this discussion.
> > Do
> > > > you
> > > > > have anything to add and improve?
> > > > > Best,
> > > > > Forward
> > > > >
> > > > > Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：
> > > > >
> > > > > > Hi Jack,
> > > > > > Thank you very much for your reply, google doc I have updated,
> and
> > > some
> > > > > of
> > > > > > your questions I replied.
> > > > > >  In addition, I want to apply for Flip permissions for this
> > purpose.
> > > > > >
> > > > > > Best,
> > > > > > Forward
> > > > > >
> > > > > > Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
> > > > > >
> > > > > >> Hi Forward,
> > > > > >>
> > > > > >> Sorry for the late reply. I have went through the design doc
> and I
> > > > think
> > > > > >> it
> > > > > >> is very nice.
> > > > > >>
> > > > > >> Here are my thoughts and suggestions:
> > > > > >>
> > > > > >> 0) I think support JSON functions in SQL is not complicated.
> > Because
> > > > > >> Calcite already supports the parser part and the runtime part.
> > > > > >>     We only need to integrate it in Flink and add good coverage
> > > tests.
> > > > > >> 1) However, I think we should also design the corresponding JSON
> > > > > Functions
> > > > > >> API for Table API which is very important.
> > > > > >>     I don't have a clear idea about how to support all the JSON
> > > > Function
> > > > > >> syntax in Table API. And this may need more discussions.
> > > > > >> 2) IMO, it deserves a FLIP (especially for the Table API part).
> > You
> > > > can
> > > > > >> follow the FLIP process [1] to start a FLIP proposal.
> > > > > >> 3) I think we only need to implement it in blink planner as we
> are
> > > > going
> > > > > >> to
> > > > > >> deprecate old planner.
> > > > > >>    So could you update the implementation section in the doc
> > because
> > > > the
> > > > > >> implementation in blink planner should be different.
> > > > > >> 4) It would be better to have an implementation plan to priority
> > the
> > > > > >> sub-tasks.
> > > > > >>     From my point of view, JSON_VALUE is the most important and
> > > > > JSON_TABLE
> > > > > >> gets the least priority.
> > > > > >>
> > > > > >> I also left some comments in the google doc.
> > > > > >>
> > > > > >> Hi @JingsongLee <[hidden email]> ,
> > > > > >>
> > > > > >> I think we don't need to wait for FLIP-51. As we don't have a
> > clear
> > > > > >> progress of FLIP-51.
> > > > > >> And as far as I know, it will add a few of PlannerExpressions
> > which
> > > > can
> > > > > be
> > > > > >> refactored easily during FLIP-51.
> > > > > >>
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Jark
> > > > > >>
> > > > > >> [1]:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]>
> > > wrote:
> > > > > >>
> > > > > >> > +1 to have JSON functions in Flink SQL
> > > > > >> >
> > > > > >> > JingsongLee <[hidden email]> 于2019年9月5日周四
> > > > 下午4:46写道：
> > > > > >> >
> > > > > >> > > +1
> > > > > >> > > Nice document. I think it is easier to do after expression
> > > > > >> reworking[1].
> > > > > >> > > By the way, which planner do you want to start?
> > > > > >> > >
> > > > > >> > > [1]
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
> > > > > >> > >
> > > > > >> > > Best,
> > > > > >> > > Jingsong Lee
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > ------------------------------------------------------------------
> > > > > >> > > From:TANG Wen-hui <[hidden email]>
> > > > > >> > > Send Time:2019年9月5日(星期四) 14:36
> > > > > >> > > To:dev <[hidden email]>
> > > > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink
> SQL
> > > > > >> > >
> > > > > >> > > +1
> > > > > >> > > I have done similar work before.
> > > > > >> > > Looking forward to discussing this feature.
> > > > > >> > >
> > > > > >> > > Best
> > > > > >> > > wenhui
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > [hidden email]
> > > > > >> > >
> > > > > >> > > From: Kurt Young
> > > > > >> > > Date: 2019-09-05 14:00
> > > > > >> > > To: dev
> > > > > >> > > CC: Anyang Hu
> > > > > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
> > > > > >> > > +1 to add JSON support to Flink. We also see lots of
> > > requirements
> > > > > for
> > > > > >> > JSON
> > > > > >> > > related functions in our internal platform. Since these are
> > > > already
> > > > > >> SQL
> > > > > >> > > standard, I think it's a good time to add them to Flink.
> > > > > >> > >
> > > > > >> > > Best,
> > > > > >> > > Kurt
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]>
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > We also see strong demands from our SQL users for
> JSON/Date
> > > > > related
> > > > > >> > > > functions.
> > > > > >> > > >
> > > > > >> > > > Also +Anyang Hu <[hidden email]>
> > > > > >> > > >
> > > > > >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]>
> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Forward,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for bringing this discussion and preparing the
> nice
> > > > > design.
> > > > > >> > > > > I think it's nice to have the JSON functions in the next
> > > > > release.
> > > > > >> > > > > We have received some requirements for this feature.
> > > > > >> > > > >
> > > > > >> > > > > I can help to shepherd this JSON functions effort and
> will
> > > > leave
> > > > > >> > > comments
> > > > > >> > > > >  in the design doc in the next days.
> > > > > >> > > > >
> > > > > >> > > > > Hi Danny,
> > > > > >> > > > >
> > > > > >> > > > > The new introduced JSON functions are from SQL:2016, not
> > > from
> > > > > >> MySQL.
> > > > > >> > > > > So there no JSON type is needed. According to the
> > SQL:2016,
> > > > the
> > > > > >> > > > > representation of JSON data can be "character string"
> > which
> > > is
> > > > > >> also
> > > > > >> > > > > the current implementation in Calcite[1].
> > > > > >> > > > >
> > > > > >> > > > > Best,
> > > > > >> > > > > Jark
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > [1]:
> > > > > >> https://calcite.apache.org/docs/reference.html#json-functions
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <
> > > > [hidden email]
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > hi Danny Chan ,Thank you very much for your reply,
> your
> > > help
> > > > > can
> > > > > >> > help
> > > > > >> > > > me
> > > > > >> > > > > > further improve this discussion.
> > > > > >> > > > > > Best
> > > > > >> > > > > > forward
> > > > > >> > > > > >
> > > > > >> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三
> > 下午8:50写道：
> > > > > >> > > > > >
> > > > > >> > > > > > > Thanks Xu Forward for bring up this topic, I think
> the
> > > > JSON
> > > > > >> > > functions
> > > > > >> > > > > are
> > > > > >> > > > > > > very useful especially for those MySQL users.
> > > > > >> > > > > > >
> > > > > >> > > > > > > I saw that you have done some work within the Apache
> > > > > Calcite,
> > > > > >> > > that’s
> > > > > >> > > > a
> > > > > >> > > > > > > good start, but this is one concern from me, Flink
> > > doesn’t
> > > > > >> > support
> > > > > >> > > > JSON
> > > > > >> > > > > > > type internal, so how to represent a JSON object in
> > > Flink
> > > > > >> maybe a
> > > > > >> > > key
> > > > > >> > > > > > point
> > > > > >> > > > > > > we need to resolve. In Calcite, we use ANY type to
> > > > represent
> > > > > >> as
> > > > > >> > the
> > > > > >> > > > > JSON,
> > > > > >> > > > > > > but I don’t think it is the right way to go, maybe
> we
> > > can
> > > > > >> have a
> > > > > >> > > > > > discussion
> > > > > >> > > > > > > here.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Best,
> > > > > >> > > > > > > Danny Chan
> > > > > >> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <
> > > > [hidden email]
> > > > > >> >，写道：
> > > > > >> > > > > > > > Hi everybody,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I'd like to kick off a discussion on Support JSON
> > > > > functions
> > > > > >> in
> > > > > >> > > > Flink
> > > > > >> > > > > > SQL.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > The entire plan is divided into two steps:
> > > > > >> > > > > > > > 1. Implement Support SQL 2016-2017 JSON functions
> in
> > > > Flink
> > > > > >> > > SQL[1].
> > > > > >> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON
> > functions
> > > in
> > > > > >> Flink
> > > > > >> > > SQL,
> > > > > >> > > > > > such
> > > > > >> > > > > > > as
> > > > > >> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful
> > JSON
> > > > > >> > functions.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Would love to hear your thoughts.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > [1]
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Best,
> > > > > >> > > > > > > > ForwardXu
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best, Jingsong Lee
> > > >
> > >
> >
>

[Jark Wu-2]

Hi Forward,

Thanks for updating the documentation. I think the design doc looks good to
me now. I think you can convert it into FLIP wiki and start a vote then.

Best,Jark

Jark Wu <[hidden email]>于2019年12月20日 周五23:07写道：

> Hi Forward,
>
> Thanks for updating the documentation. I think it is
>
> On Mon, 2 Dec 2019 at 14:21, Forward Xu <[hidden email]> wrote:
>
>> Hi Jark,
>> Thank you very much，I will improve this document as soon as possible.
>> My confluence username is ForwardXu.
>>
>> Best,
>> Forward
>>
>> Jark Wu <[hidden email]> 于2019年12月2日周一 上午11:56写道：
>>
>> > Hi Forward,
>> >
>> > Sorry for the late reply.
>> > As I said before, it would be better to include the JSON functions API
>> for
>> > Table API.
>> > For example, what the "json_exist", "json_value", "json_query", etc...
>> > functions in Table API looks like.
>> > And it would be better to follow FLIP template [1] which include "Public
>> > Interface" (brief list of public interface), and "Proposed changes"
>> > (detailed proposal).
>> >
>> > Once the design doc looks good, you can update it to FLIP and start a
>> vote.
>> >
>> > Btw, what's your username in confluence?
>> >
>> >
>> > Best,
>> > Jark
>> >
>> >
>> > [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template
>> >
>> >
>> > On Mon, 2 Dec 2019 at 11:44, Forward Xu <[hidden email]> wrote:
>> >
>> > > Hi Jingsong Lee :
>> > >
>> > > Thank you very much, I need to apply for FLIP permission. Do I need to
>> > > create a FLIP for this?
>> > >
>> > > Best,
>> > > Forward
>> > >
>> > > Jingsong Li <[hidden email]> 于2019年12月2日周一 上午11:40写道：
>> > >
>> > > > Hi Forward:
>> > > >
>> > > > Document looks good to me.
>> > > > I think you can just start doing this.
>> > > > They all work very independently, so I don't think there's any
>> obvious
>> > > > blocking.
>> > > >
>> > > > Best,
>> > > > Jingsong Lee
>> > > >
>> > > > On Sat, Nov 30, 2019 at 10:59 AM Forward Xu <[hidden email]
>> >
>> > > > wrote:
>> > > >
>> > > > > Hi everyone, It's been a long time since I started this
>> discussion.
>> > Do
>> > > > you
>> > > > > have anything to add and improve?
>> > > > > Best,
>> > > > > Forward
>> > > > >
>> > > > > Forward Xu <[hidden email]> 于2019年9月22日周日 下午6:30写道：
>> > > > >
>> > > > > > Hi Jack,
>> > > > > > Thank you very much for your reply, google doc I have updated,
>> and
>> > > some
>> > > > > of
>> > > > > > your questions I replied.
>> > > > > >  In addition, I want to apply for Flip permissions for this
>> > purpose.
>> > > > > >
>> > > > > > Best,
>> > > > > > Forward
>> > > > > >
>> > > > > > Jark Wu <[hidden email]> 于2019年9月20日周五 下午9:53写道：
>> > > > > >
>> > > > > >> Hi Forward,
>> > > > > >>
>> > > > > >> Sorry for the late reply. I have went through the design doc
>> and I
>> > > > think
>> > > > > >> it
>> > > > > >> is very nice.
>> > > > > >>
>> > > > > >> Here are my thoughts and suggestions:
>> > > > > >>
>> > > > > >> 0) I think support JSON functions in SQL is not complicated.
>> > Because
>> > > > > >> Calcite already supports the parser part and the runtime part.
>> > > > > >>     We only need to integrate it in Flink and add good coverage
>> > > tests.
>> > > > > >> 1) However, I think we should also design the corresponding
>> JSON
>> > > > > Functions
>> > > > > >> API for Table API which is very important.
>> > > > > >>     I don't have a clear idea about how to support all the JSON
>> > > > Function
>> > > > > >> syntax in Table API. And this may need more discussions.
>> > > > > >> 2) IMO, it deserves a FLIP (especially for the Table API part).
>> > You
>> > > > can
>> > > > > >> follow the FLIP process [1] to start a FLIP proposal.
>> > > > > >> 3) I think we only need to implement it in blink planner as we
>> are
>> > > > going
>> > > > > >> to
>> > > > > >> deprecate old planner.
>> > > > > >>    So could you update the implementation section in the doc
>> > because
>> > > > the
>> > > > > >> implementation in blink planner should be different.
>> > > > > >> 4) It would be better to have an implementation plan to
>> priority
>> > the
>> > > > > >> sub-tasks.
>> > > > > >>     From my point of view, JSON_VALUE is the most important and
>> > > > > JSON_TABLE
>> > > > > >> gets the least priority.
>> > > > > >>
>> > > > > >> I also left some comments in the google doc.
>> > > > > >>
>> > > > > >> Hi @JingsongLee <[hidden email]> ,
>> > > > > >>
>> > > > > >> I think we don't need to wait for FLIP-51. As we don't have a
>> > clear
>> > > > > >> progress of FLIP-51.
>> > > > > >> And as far as I know, it will add a few of PlannerExpressions
>> > which
>> > > > can
>> > > > > be
>> > > > > >> refactored easily during FLIP-51.
>> > > > > >>
>> > > > > >>
>> > > > > >> Cheers,
>> > > > > >> Jark
>> > > > > >>
>> > > > > >> [1]:
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> On Thu, 5 Sep 2019 at 19:29, vino yang <[hidden email]>
>> > > wrote:
>> > > > > >>
>> > > > > >> > +1 to have JSON functions in Flink SQL
>> > > > > >> >
>> > > > > >> > JingsongLee <[hidden email]> 于2019年9月5日周四
>> > > > 下午4:46写道：
>> > > > > >> >
>> > > > > >> > > +1
>> > > > > >> > > Nice document. I think it is easier to do after expression
>> > > > > >> reworking[1].
>> > > > > >> > > By the way, which planner do you want to start?
>> > > > > >> > >
>> > > > > >> > > [1]
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
>> > > > > >> > >
>> > > > > >> > > Best,
>> > > > > >> > > Jingsong Lee
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > >
>> > > ------------------------------------------------------------------
>> > > > > >> > > From:TANG Wen-hui <[hidden email]>
>> > > > > >> > > Send Time:2019年9月5日(星期四) 14:36
>> > > > > >> > > To:dev <[hidden email]>
>> > > > > >> > > Subject:Re: Re: [DISCUSS] Support JSON functions in Flink
>> SQL
>> > > > > >> > >
>> > > > > >> > > +1
>> > > > > >> > > I have done similar work before.
>> > > > > >> > > Looking forward to discussing this feature.
>> > > > > >> > >
>> > > > > >> > > Best
>> > > > > >> > > wenhui
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > [hidden email]
>> > > > > >> > >
>> > > > > >> > > From: Kurt Young
>> > > > > >> > > Date: 2019-09-05 14:00
>> > > > > >> > > To: dev
>> > > > > >> > > CC: Anyang Hu
>> > > > > >> > > Subject: Re: [DISCUSS] Support JSON functions in Flink SQL
>> > > > > >> > > +1 to add JSON support to Flink. We also see lots of
>> > > requirements
>> > > > > for
>> > > > > >> > JSON
>> > > > > >> > > related functions in our internal platform. Since these are
>> > > > already
>> > > > > >> SQL
>> > > > > >> > > standard, I think it's a good time to add them to Flink.
>> > > > > >> > >
>> > > > > >> > > Best,
>> > > > > >> > > Kurt
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > On Thu, Sep 5, 2019 at 10:37 AM Qi Luo <[hidden email]
>> >
>> > > > wrote:
>> > > > > >> > >
>> > > > > >> > > > We also see strong demands from our SQL users for
>> JSON/Date
>> > > > > related
>> > > > > >> > > > functions.
>> > > > > >> > > >
>> > > > > >> > > > Also +Anyang Hu <[hidden email]>
>> > > > > >> > > >
>> > > > > >> > > > On Wed, Sep 4, 2019 at 9:51 PM Jark Wu <[hidden email]
>> >
>> > > > wrote:
>> > > > > >> > > >
>> > > > > >> > > > > Hi Forward,
>> > > > > >> > > > >
>> > > > > >> > > > > Thanks for bringing this discussion and preparing the
>> nice
>> > > > > design.
>> > > > > >> > > > > I think it's nice to have the JSON functions in the
>> next
>> > > > > release.
>> > > > > >> > > > > We have received some requirements for this feature.
>> > > > > >> > > > >
>> > > > > >> > > > > I can help to shepherd this JSON functions effort and
>> will
>> > > > leave
>> > > > > >> > > comments
>> > > > > >> > > > >  in the design doc in the next days.
>> > > > > >> > > > >
>> > > > > >> > > > > Hi Danny,
>> > > > > >> > > > >
>> > > > > >> > > > > The new introduced JSON functions are from SQL:2016,
>> not
>> > > from
>> > > > > >> MySQL.
>> > > > > >> > > > > So there no JSON type is needed. According to the
>> > SQL:2016,
>> > > > the
>> > > > > >> > > > > representation of JSON data can be "character string"
>> > which
>> > > is
>> > > > > >> also
>> > > > > >> > > > > the current implementation in Calcite[1].
>> > > > > >> > > > >
>> > > > > >> > > > > Best,
>> > > > > >> > > > > Jark
>> > > > > >> > > > >
>> > > > > >> > > > >
>> > > > > >> > > > > [1]:
>> > > > > >> https://calcite.apache.org/docs/reference.html#json-functions
>> > > > > >> > > > >
>> > > > > >> > > > >
>> > > > > >> > > > > On Wed, 4 Sep 2019 at 21:22, Xu Forward <
>> > > > [hidden email]
>> > > > > >
>> > > > > >> > > wrote:
>> > > > > >> > > > >
>> > > > > >> > > > > > hi Danny Chan ,Thank you very much for your reply,
>> your
>> > > help
>> > > > > can
>> > > > > >> > help
>> > > > > >> > > > me
>> > > > > >> > > > > > further improve this discussion.
>> > > > > >> > > > > > Best
>> > > > > >> > > > > > forward
>> > > > > >> > > > > >
>> > > > > >> > > > > > Danny Chan <[hidden email]> 于2019年9月4日周三
>> > 下午8:50写道：
>> > > > > >> > > > > >
>> > > > > >> > > > > > > Thanks Xu Forward for bring up this topic, I think
>> the
>> > > > JSON
>> > > > > >> > > functions
>> > > > > >> > > > > are
>> > > > > >> > > > > > > very useful especially for those MySQL users.
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > I saw that you have done some work within the
>> Apache
>> > > > > Calcite,
>> > > > > >> > > that’s
>> > > > > >> > > > a
>> > > > > >> > > > > > > good start, but this is one concern from me, Flink
>> > > doesn’t
>> > > > > >> > support
>> > > > > >> > > > JSON
>> > > > > >> > > > > > > type internal, so how to represent a JSON object in
>> > > Flink
>> > > > > >> maybe a
>> > > > > >> > > key
>> > > > > >> > > > > > point
>> > > > > >> > > > > > > we need to resolve. In Calcite, we use ANY type to
>> > > > represent
>> > > > > >> as
>> > > > > >> > the
>> > > > > >> > > > > JSON,
>> > > > > >> > > > > > > but I don’t think it is the right way to go, maybe
>> we
>> > > can
>> > > > > >> have a
>> > > > > >> > > > > > discussion
>> > > > > >> > > > > > > here.
>> > > > > >> > > > > > >
>> > > > > >> > > > > > > Best,
>> > > > > >> > > > > > > Danny Chan
>> > > > > >> > > > > > > 在 2019年9月4日 +0800 PM8:34，Xu Forward <
>> > > > [hidden email]
>> > > > > >> >，写道：
>> > > > > >> > > > > > > > Hi everybody,
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > I'd like to kick off a discussion on Support JSON
>> > > > > functions
>> > > > > >> in
>> > > > > >> > > > Flink
>> > > > > >> > > > > > SQL.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > The entire plan is divided into two steps:
>> > > > > >> > > > > > > > 1. Implement Support SQL 2016-2017 JSON
>> functions in
>> > > > Flink
>> > > > > >> > > SQL[1].
>> > > > > >> > > > > > > > 2. Implement non-Support SQL 2016-2017 JSON
>> > functions
>> > > in
>> > > > > >> Flink
>> > > > > >> > > SQL,
>> > > > > >> > > > > > such
>> > > > > >> > > > > > > as
>> > > > > >> > > > > > > > JSON_TYPE in Mysql, JSON_LENGTH, etc. Very useful
>> > JSON
>> > > > > >> > functions.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > Would love to hear your thoughts.
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > [1]
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp
>> > > > > >> > > > > > > >
>> > > > > >> > > > > > > > Best,
>> > > > > >> > > > > > > > ForwardXu
>> > > > > >> > > > > > >
>> > > > > >> > > > > >
>> > > > > >> > > > >
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best, Jingsong Lee
>> > > >
>> > >
>> >
>>
>

[Forward Xu]

Hi everybody,


I'd like to kick off a discussion on FLIP-90: Support SQL 2016-2017 JSON
functions in Flink SQL.


Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].



Would love to hear your thoughts.


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-90%3A+Support+SQL+2016-2017+JSON+functions+in+Flink+SQL
<https://docs.google.com/document/d/1JfaFYIFOAY8P2pFhOYNCQ9RTzwF4l85_bnTvImOLKMk/edit#heading=h.76mb88ca6yjp>


Best,

ForwardXu

>

[Forward Xu]

Hi everybody,


I'd like to kick off a discussion on FLIP-90: Support SQL 2016-2017 JSON
functions in Flink SQL.


Implement Support SQL 2016-2017 JSON functions in Flink SQL[1].



Would love to hear your thoughts.


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-90%3A+Support+SQL+2016-2017+JSON+functions+in+Flink+SQL


Best,

ForwardXu

>