[DISCUSS] FLIP-114: Support Python UDF in SQL Client

[Wei Zhong-2]

Hi everyone,

I would like to start discussion about how to support Python UDF in SQL Client.

Flink Python UDF(FLIP-58[1]) has already been introduced in the release of 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].  

SQL Client defines UDF via the environment file and has its own CLI implementation to manage dependencies, but neither of which supports Python UDF. We want to introduce the support of Python UDF for SQL Client, including the registration and the dependency management of Python UDF.

Here is the design doc:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client

Looking forward to your feedback!

Best,
Wei

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL

[godfreyhe]

Hi Wei, thanks for the proposal.

I think it's better to give two more examples, one is how to use python UDF
in SQL, another is how to start sql-client.sh with full python dependencies.

Best,
Godfrey

Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：

> Hi everyone,
>
> I would like to start discussion about how to support Python UDF in SQL
> Client.
>
> Flink Python UDF(FLIP-58[1]) has already been introduced in the release of
> 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
>
> SQL Client defines UDF via the environment file and has its own CLI
> implementation to manage dependencies, but neither of which supports Python
> UDF. We want to introduce the support of Python UDF for SQL Client,
> including the registration and the dependency management of Python UDF.
>
> Here is the design doc:
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
>
> Looking forward to your feedback!
>
> Best,
> Wei
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
>
>

[Xingbo Huang]

Hi Wei,
Thanks a lot for the proposal.


Best,
Xingbo

godfrey he <[hidden email]> 于2020年3月9日周一 下午10:24写道：

> Hi Wei, thanks for the proposal.
>
> I think it's better to give two more examples, one is how to use python UDF
> in SQL, another is how to start sql-client.sh with full python
> dependencies.
>
> Best,
> Godfrey
>
> Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：
>
> > Hi everyone,
> >
> > I would like to start discussion about how to support Python UDF in SQL
> > Client.
> >
> > Flink Python UDF(FLIP-58[1]) has already been introduced in the release
> of
> > 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
> >
> > SQL Client defines UDF via the environment file and has its own CLI
> > implementation to manage dependencies, but neither of which supports
> Python
> > UDF. We want to introduce the support of Python UDF for SQL Client,
> > including the registration and the dependency management of Python UDF.
> >
> > Here is the design doc:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> >
> > Looking forward to your feedback!
> >
> > Best,
> > Wei
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >
> >
>

[Xingbo Huang]

Hi Godfrey,
thanks for your suggestion.
I have added two examples how to use python UDF
in SQL and how to start sql-client.sh with full python dependencies In FLIP.

Best,
Xingo

godfrey he <[hidden email]> 于2020年3月9日周一 下午10:24写道：

> Hi Wei, thanks for the proposal.
>
> I think it's better to give two more examples, one is how to use python UDF
> in SQL, another is how to start sql-client.sh with full python
> dependencies.
>
> Best,
> Godfrey
>
> Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：
>
> > Hi everyone,
> >
> > I would like to start discussion about how to support Python UDF in SQL
> > Client.
> >
> > Flink Python UDF(FLIP-58[1]) has already been introduced in the release
> of
> > 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
> >
> > SQL Client defines UDF via the environment file and has its own CLI
> > implementation to manage dependencies, but neither of which supports
> Python
> > UDF. We want to introduce the support of Python UDF for SQL Client,
> > including the registration and the dependency management of Python UDF.
> >
> > Here is the design doc:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> >
> > Looking forward to your feedback!
> >
> > Best,
> > Wei
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >
> >
>

[Wei Zhong-2]

Hi everyone,

Are there more comments about this FLIP? If not, I would like to bring up the VOTE.

Best,
Wei

> 在 2020年3月9日，23:18，Xingbo Huang <[hidden email]> 写道：
>
> Hi Godfrey,
> thanks for your suggestion.
> I have added two examples how to use python UDF
> in SQL and how to start sql-client.sh with full python dependencies In FLIP.
>
> Best,
> Xingo
>
> godfrey he <[hidden email]> 于2020年3月9日周一 下午10:24写道：
>
>> Hi Wei, thanks for the proposal.
>>
>> I think it's better to give two more examples, one is how to use python UDF
>> in SQL, another is how to start sql-client.sh with full python
>> dependencies.
>>
>> Best,
>> Godfrey
>>
>> Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：
>>
>>> Hi everyone,
>>>
>>> I would like to start discussion about how to support Python UDF in SQL
>>> Client.
>>>
>>> Flink Python UDF(FLIP-58[1]) has already been introduced in the release
>> of
>>> 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
>>>
>>> SQL Client defines UDF via the environment file and has its own CLI
>>> implementation to manage dependencies, but neither of which supports
>> Python
>>> UDF. We want to introduce the support of Python UDF for SQL Client,
>>> including the registration and the dependency management of Python UDF.
>>>
>>> Here is the design doc:
>>>
>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
>>>
>>> Looking forward to your feedback!
>>>
>>> Best,
>>> Wei
>>>
>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>> [2]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
>>>
>>>
>>

[jincheng sun]

Hi Wei,

+1, Thanks for this discussion which is crucial for SQL users to use
PyFlink. Would be great to bring up the VOTE thread.

Best,
Jincheng


Wei Zhong <[hidden email]> 于2020年3月30日周一 下午2:38写道：

> Hi everyone,
>
> Are there more comments about this FLIP? If not, I would like to bring up
> the VOTE.
>
> Best,
> Wei
>
> > 在 2020年3月9日，23:18，Xingbo Huang <[hidden email]> 写道：
> >
> > Hi Godfrey,
> > thanks for your suggestion.
> > I have added two examples how to use python UDF
> > in SQL and how to start sql-client.sh with full python dependencies In
> FLIP.
> >
> > Best,
> > Xingo
> >
> > godfrey he <[hidden email]> 于2020年3月9日周一 下午10:24写道：
> >
> >> Hi Wei, thanks for the proposal.
> >>
> >> I think it's better to give two more examples, one is how to use python
> UDF
> >> in SQL, another is how to start sql-client.sh with full python
> >> dependencies.
> >>
> >> Best,
> >> Godfrey
> >>
> >> Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：
> >>
> >>> Hi everyone,
> >>>
> >>> I would like to start discussion about how to support Python UDF in SQL
> >>> Client.
> >>>
> >>> Flink Python UDF(FLIP-58[1]) has already been introduced in the release
> >> of
> >>> 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
> >>>
> >>> SQL Client defines UDF via the environment file and has its own CLI
> >>> implementation to manage dependencies, but neither of which supports
> >> Python
> >>> UDF. We want to introduce the support of Python UDF for SQL Client,
> >>> including the registration and the dependency management of Python UDF.
> >>>
> >>> Here is the design doc:
> >>>
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> >>>
> >>> Looking forward to your feedback!
> >>>
> >>> Best,
> >>> Wei
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> >>> [2]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >>>
> >>>
> >>
>
>

[Dian Fu-2]

Thanks Wei for this work! +1 to bring up the VOTE thread.

> 在 2020年3月30日，下午2:43，jincheng sun <[hidden email]> 写道：
>
> Hi Wei,
>
> +1, Thanks for this discussion which is crucial for SQL users to use
> PyFlink. Would be great to bring up the VOTE thread.
>
> Best,
> Jincheng
>
>
> Wei Zhong <[hidden email]> 于2020年3月30日周一 下午2:38写道：
>
>> Hi everyone,
>>
>> Are there more comments about this FLIP? If not, I would like to bring up
>> the VOTE.
>>
>> Best,
>> Wei
>>
>>> 在 2020年3月9日，23:18，Xingbo Huang <[hidden email]> 写道：
>>>
>>> Hi Godfrey,
>>> thanks for your suggestion.
>>> I have added two examples how to use python UDF
>>> in SQL and how to start sql-client.sh with full python dependencies In
>> FLIP.
>>>
>>> Best,
>>> Xingo
>>>
>>> godfrey he <[hidden email]> 于2020年3月9日周一 下午10:24写道：
>>>
>>>> Hi Wei, thanks for the proposal.
>>>>
>>>> I think it's better to give two more examples, one is how to use python
>> UDF
>>>> in SQL, another is how to start sql-client.sh with full python
>>>> dependencies.
>>>>
>>>> Best,
>>>> Godfrey
>>>>
>>>> Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I would like to start discussion about how to support Python UDF in SQL
>>>>> Client.
>>>>>
>>>>> Flink Python UDF(FLIP-58[1]) has already been introduced in the release
>>>> of
>>>>> 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
>>>>>
>>>>> SQL Client defines UDF via the environment file and has its own CLI
>>>>> implementation to manage dependencies, but neither of which supports
>>>> Python
>>>>> UDF. We want to introduce the support of Python UDF for SQL Client,
>>>>> including the registration and the dependency management of Python UDF.
>>>>>
>>>>> Here is the design doc:
>>>>>
>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
>>>>>
>>>>> Looking forward to your feedback!
>>>>>
>>>>> Best,
>>>>> Wei
>>>>>
>>>>> [1]
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>>>> [2]
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
>>>>>
>>>>>
>>>>
>>
>>

[Hequn Cheng-2]

Hi Wei,

Thanks a lot for the proposal! +1 for the VOTE.

Best,
Hequn



On Mon, Mar 30, 2020 at 3:31 PM Dian Fu <[hidden email]> wrote:

> Thanks Wei for this work! +1 to bring up the VOTE thread.
>
> > 在 2020年3月30日，下午2:43，jincheng sun <[hidden email]> 写道：
> >
> > Hi Wei,
> >
> > +1, Thanks for this discussion which is crucial for SQL users to use
> > PyFlink. Would be great to bring up the VOTE thread.
> >
> > Best,
> > Jincheng
> >
> >
> > Wei Zhong <[hidden email]> 于2020年3月30日周一 下午2:38写道：
> >
> >> Hi everyone,
> >>
> >> Are there more comments about this FLIP? If not, I would like to bring
> up
> >> the VOTE.
> >>
> >> Best,
> >> Wei
> >>
> >>> 在 2020年3月9日，23:18，Xingbo Huang <[hidden email]> 写道：
> >>>
> >>> Hi Godfrey,
> >>> thanks for your suggestion.
> >>> I have added two examples how to use python UDF
> >>> in SQL and how to start sql-client.sh with full python dependencies In
> >> FLIP.
> >>>
> >>> Best,
> >>> Xingo
> >>>
> >>> godfrey he <[hidden email]> 于2020年3月9日周一 下午10:24写道：
> >>>
> >>>> Hi Wei, thanks for the proposal.
> >>>>
> >>>> I think it's better to give two more examples, one is how to use
> python
> >> UDF
> >>>> in SQL, another is how to start sql-client.sh with full python
> >>>> dependencies.
> >>>>
> >>>> Best,
> >>>> Godfrey
> >>>>
> >>>> Wei Zhong <[hidden email]> 于2020年3月9日周一 下午10:09写道：
> >>>>
> >>>>> Hi everyone,
> >>>>>
> >>>>> I would like to start discussion about how to support Python UDF in
> SQL
> >>>>> Client.
> >>>>>
> >>>>> Flink Python UDF(FLIP-58[1]) has already been introduced in the
> release
> >>>> of
> >>>>> 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
> >>>>>
> >>>>> SQL Client defines UDF via the environment file and has its own CLI
> >>>>> implementation to manage dependencies, but neither of which supports
> >>>> Python
> >>>>> UDF. We want to introduce the support of Python UDF for SQL Client,
> >>>>> including the registration and the dependency management of Python
> UDF.
> >>>>>
> >>>>> Here is the design doc:
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> >>>>>
> >>>>> Looking forward to your feedback!
> >>>>>
> >>>>> Best,
> >>>>> Wei
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> >>>>> [2]
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>