[DISCUSS] Flink Python UDF Environment and Dependency Management

[Wei Zhong-2]

Hi everyone,

In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of python UDF, the environment and dependency management of users' python code has not been fully discussed.

I'd like to start a discussion on "Flink Python UDF Environment and Dependency Management". Here is the design doc I drafted:

https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing <https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing>

Please take a look, and feedbacks are welcome.

Thanks,
Wei

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>

[jincheng sun]

Thanks for bring up the discussion, Wei.
Overall the design doc looks good. I have left a few comments.

BTW: Dependency Management is very important for Python UDFs, welcome
anyone left your suggestions!

Best,
Jincheng

Wei Zhong <[hidden email]> 于2019年9月26日周四 上午11:59写道：

> Hi everyone,
>
> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
> python UDF, the environment and dependency management of users' python code
> has not been fully discussed.
>
> I'd like to start a discussion on "Flink Python UDF Environment and
> Dependency Management". Here is the design doc I drafted:
>
>
> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>
> Please take a look, and feedbacks are welcome.
>
> Thanks,
> Wei
>
> [1]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>
>
>

[Dian Fu-2]

Hi Wei,

Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.

Thanks,
Dian

> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email]> 写道：
>
> Thanks for bring up the discussion, Wei.
> Overall the design doc looks good. I have left a few comments.
>
> BTW: Dependency Management is very important for Python UDFs, welcome
> anyone left your suggestions!
>
> Best,
> Jincheng
>
> Wei Zhong <[hidden email]> 于2019年9月26日周四 上午11:59写道：
>
>> Hi everyone,
>>
>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>> python UDF, the environment and dependency management of users' python code
>> has not been fully discussed.
>>
>> I'd like to start a discussion on "Flink Python UDF Environment and
>> Dependency Management". Here is the design doc I drafted:
>>
>>
>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>>
>> Please take a look, and feedbacks are welcome.
>>
>> Thanks,
>> Wei
>>
>> [1]:
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>
>>
>>

[Wei Zhong-2]

Hi Jincheng, Dian and Jeff,

Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
  - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
  - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.

I'm also glad to hear any other opinions!

Thanks,
Wei

> 在 2019年9月26日，15:23，Dian Fu <[hidden email]> 写道：
>
> Hi Wei,
>
> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
>
> Thanks,
> Dian
>
>> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email]> 写道：
>>
>> Thanks for bring up the discussion, Wei.
>> Overall the design doc looks good. I have left a few comments.
>>
>> BTW: Dependency Management is very important for Python UDFs, welcome
>> anyone left your suggestions!
>>
>> Best,
>> Jincheng
>>
>> Wei Zhong <[hidden email]> 于2019年9月26日周四 上午11:59写道：
>>
>>> Hi everyone,
>>>
>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>>> python UDF, the environment and dependency management of users' python code
>>> has not been fully discussed.
>>>
>>> I'd like to start a discussion on "Flink Python UDF Environment and
>>> Dependency Management". Here is the design doc I drafted:
>>>
>>>
>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>>>
>>> Please take a look, and feedbacks are welcome.
>>>
>>> Thanks,
>>> Wei
>>>
>>> [1]:
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>
>>>
>>>
>

[Dian Fu-2]

Hi Wei,

Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE.

Thanks,
Dian

> 在 2019年10月9日，上午11:23，Wei Zhong <[hidden email]> 写道：
>
> Hi Jincheng, Dian and Jeff,
>
> Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
> - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
> - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
>
> I'm also glad to hear any other opinions!
>
> Thanks,
> Wei
>
>
>> 在 2019年9月26日，15:23，Dian Fu <[hidden email]> 写道：
>>
>> Hi Wei,
>>
>> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
>>
>> Thanks,
>> Dian
>>
>>> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email]> 写道：
>>>
>>> Thanks for bring up the discussion, Wei.
>>> Overall the design doc looks good. I have left a few comments.
>>>
>>> BTW: Dependency Management is very important for Python UDFs, welcome
>>> anyone left your suggestions!
>>>
>>> Best,
>>> Jincheng
>>>
>>> Wei Zhong <[hidden email]> 于2019年9月26日周四 上午11:59写道：
>>>
>>>> Hi everyone,
>>>>
>>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>>>> python UDF, the environment and dependency management of users' python code
>>>> has not been fully discussed.
>>>>
>>>> I'd like to start a discussion on "Flink Python UDF Environment and
>>>> Dependency Management". Here is the design doc I drafted:
>>>>
>>>>
>>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>>>>
>>>> Please take a look, and feedbacks are welcome.
>>>>
>>>> Thanks,
>>>> Wei
>>>>
>>>> [1]:
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>
>>>>
>>>>
>>
>

[jincheng sun]

Hi,

+ 1 to bring up the VOTE and create the FLIP.

Best,
Jincheng

Dian Fu <[hidden email]> 于2019年10月12日周六 上午10:12写道：

> Hi Wei,
>
> Thanks for the great work! It seems that it has reached an agreement on
> the design. Should we start VOTE on this design? I'm also wondering if a
> FLIP is deserved as it introduces user facing API. If so, we should create
> a FLIP before VOTE.
>
> Thanks,
> Dian
>
> > 在 2019年10月9日，上午11:23，Wei Zhong <[hidden email]> 写道：
> >
> > Hi Jincheng, Dian and Jeff,
> >
> > Thank you for your replies and comments in google doc! I think we have
> come to an agreement on the desgin doc with only minor changes as follow:
> > - Using the API "set_python_executable" instead of
> "set_environment_variable" to set the python executable file path.
> > - Making the argument "requirements_cached_dir" of API
> "set_python_requirements" optional to support only upload a requirement.txt
> file.
> >
> > I'm also glad to hear any other opinions!
> >
> > Thanks,
> > Wei
> >
> >
> >> 在 2019年9月26日，15:23，Dian Fu <[hidden email]> 写道：
> >>
> >> Hi Wei,
> >>
> >> Thanks a lot for bringing up this discussion. Python dependency
> management is very important for Python users. I have left a few comments
> on the design doc.
> >>
> >> Thanks,
> >> Dian
> >>
> >>> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email]> 写道：
> >>>
> >>> Thanks for bring up the discussion, Wei.
> >>> Overall the design doc looks good. I have left a few comments.
> >>>
> >>> BTW: Dependency Management is very important for Python UDFs, welcome
> >>> anyone left your suggestions!
> >>>
> >>> Best,
> >>> Jincheng
> >>>
> >>> Wei Zhong <[hidden email]> 于2019年9月26日周四 上午11:59写道：
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical
> part of
> >>>> python UDF, the environment and dependency management of users'
> python code
> >>>> has not been fully discussed.
> >>>>
> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
> >>>> Dependency Management". Here is the design doc I drafted:
> >>>>
> >>>>
> >>>>
> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
> >>>>
> >>>> Please take a look, and feedbacks are welcome.
> >>>>
> >>>> Thanks,
> >>>> Wei
> >>>>
> >>>> [1]:
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> >>>> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table
> >
> >>>>
> >>>>
> >>
> >
>
>

[Wei Zhong-2]

Hi Jincheng,

As communicated in this email thread, I’m proposing to convert the design doc to a FLIP and bring up the VOTE. It would be great if you can grant me the write access to Confluence :). My Confluence ID is zhongwei.

Thanks,
Wei Zhong

> 在 2019年10月12日，17:41，jincheng sun <[hidden email]> 写道：
>
> Hi,
>
> + 1 to bring up the VOTE and create the FLIP.
>
> Best,
> Jincheng
>
> Dian Fu <[hidden email] <mailto:[hidden email]>> 于2019年10月12日周六 上午10:12写道：
> Hi Wei,
>
> Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE.
>
> Thanks,
> Dian
>
> > 在 2019年10月9日，上午11:23，Wei Zhong <[hidden email] <mailto:[hidden email]>> 写道：
> >
> > Hi Jincheng, Dian and Jeff,
> >
> > Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
> > - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
> > - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
> >
> > I'm also glad to hear any other opinions!
> >
> > Thanks,
> > Wei
> >
> >
> >> 在 2019年9月26日，15:23，Dian Fu <[hidden email] <mailto:[hidden email]>> 写道：
> >>
> >> Hi Wei,
> >>
> >> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
> >>
> >> Thanks,
> >> Dian
> >>
> >>> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email] <mailto:[hidden email]>> 写道：
> >>>
> >>> Thanks for bring up the discussion, Wei.
> >>> Overall the design doc looks good. I have left a few comments.
> >>>
> >>> BTW: Dependency Management is very important for Python UDFs, welcome
> >>> anyone left your suggestions!
> >>>
> >>> Best,
> >>> Jincheng
> >>>
> >>> Wei Zhong <[hidden email] <mailto:[hidden email]>> 于2019年9月26日周四 上午11:59写道：
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
> >>>> python UDF, the environment and dependency management of users' python code
> >>>> has not been fully discussed.
> >>>>
> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
> >>>> Dependency Management". Here is the design doc I drafted:
> >>>>
> >>>>
> >>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing <https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing>
> >>>>
> >>>> Please take a look, and feedbacks are welcome.
> >>>>
> >>>> Thanks,
> >>>> Wei
> >>>>
> >>>> [1]:
> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table>
> >>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>>
> >>>>
> >>>>
> >>
> >
>

[jincheng sun]

Hi Wei Zhong,

I have given you edit permission, could you please re-login and check it. :)

Best,
Jincheng


Wei Zhong <[hidden email]> 于2019年10月12日周六 下午6:06写道：

> Hi Jincheng,
>
> As communicated in this email thread, I’m proposing to convert the design
> doc to a FLIP and bring up the VOTE. It would be great if you can grant me
> the write access to Confluence :). My Confluence ID is zhongwei.
>
> Thanks,
> Wei Zhong
>
>
> 在 2019年10月12日，17:41，jincheng sun <[hidden email]> 写道：
>
> Hi,
>
> + 1 to bring up the VOTE and create the FLIP.
>
> Best,
> Jincheng
>
> Dian Fu <[hidden email]> 于2019年10月12日周六 上午10:12写道：
>
>> Hi Wei,
>>
>> Thanks for the great work! It seems that it has reached an agreement on
>> the design. Should we start VOTE on this design? I'm also wondering if a
>> FLIP is deserved as it introduces user facing API. If so, we should create
>> a FLIP before VOTE.
>>
>> Thanks,
>> Dian
>>
>> > 在 2019年10月9日，上午11:23，Wei Zhong <[hidden email]> 写道：
>> >
>> > Hi Jincheng, Dian and Jeff,
>> >
>> > Thank you for your replies and comments in google doc! I think we have
>> come to an agreement on the desgin doc with only minor changes as follow:
>> > - Using the API "set_python_executable" instead of
>> "set_environment_variable" to set the python executable file path.
>> > - Making the argument "requirements_cached_dir" of API
>> "set_python_requirements" optional to support only upload a requirement.txt
>> file.
>> >
>> > I'm also glad to hear any other opinions!
>> >
>> > Thanks,
>> > Wei
>> >
>> >
>> >> 在 2019年9月26日，15:23，Dian Fu <[hidden email]> 写道：
>> >>
>> >> Hi Wei,
>> >>
>> >> Thanks a lot for bringing up this discussion. Python dependency
>> management is very important for Python users. I have left a few comments
>> on the design doc.
>> >>
>> >> Thanks,
>> >> Dian
>> >>
>> >>> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email]> 写道：
>> >>>
>> >>> Thanks for bring up the discussion, Wei.
>> >>> Overall the design doc looks good. I have left a few comments.
>> >>>
>> >>> BTW: Dependency Management is very important for Python UDFs, welcome
>> >>> anyone left your suggestions!
>> >>>
>> >>> Best,
>> >>> Jincheng
>> >>>
>> >>> Wei Zhong <[hidden email]> 于2019年9月26日周四 上午11:59写道：
>> >>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical
>> part of
>> >>>> python UDF, the environment and dependency management of users'
>> python code
>> >>>> has not been fully discussed.
>> >>>>
>> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
>> >>>> Dependency Management". Here is the design doc I drafted:
>> >>>>
>> >>>>
>> >>>>
>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>> >>>>
>> >>>> Please take a look, and feedbacks are welcome.
>> >>>>
>> >>>> Thanks,
>> >>>> Wei
>> >>>>
>> >>>> [1]:
>> >>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>> >>>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table
>> >
>> >>>>
>> >>>>
>> >>
>> >
>>
>>
>

[Wei Zhong-2]

Thank you Jincheng. I have got the permission.

> 在 2019年10月12日，18:09，jincheng sun <[hidden email]> 写道：
>
> Hi Wei Zhong,
>
> I have given you edit permission, could you please re-login and check it. :)
>
> Best,
> Jincheng
>
>
> Wei Zhong <[hidden email] <mailto:[hidden email]>> 于2019年10月12日周六 下午6:06写道：
> Hi Jincheng,
>
> As communicated in this email thread, I’m proposing to convert the design doc to a FLIP and bring up the VOTE. It would be great if you can grant me the write access to Confluence :). My Confluence ID is zhongwei.
>
> Thanks,
> Wei Zhong
>
>
>> 在 2019年10月12日，17:41，jincheng sun <[hidden email] <mailto:[hidden email]>> 写道：
>>
>> Hi,
>>
>> + 1 to bring up the VOTE and create the FLIP.
>>
>> Best,
>> Jincheng
>>
>> Dian Fu <[hidden email] <mailto:[hidden email]>> 于2019年10月12日周六 上午10:12写道：
>> Hi Wei,
>>
>> Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE.
>>
>> Thanks,
>> Dian
>>
>> > 在 2019年10月9日，上午11:23，Wei Zhong <[hidden email] <mailto:[hidden email]>> 写道：
>> >
>> > Hi Jincheng, Dian and Jeff,
>> >
>> > Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
>> > - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
>> > - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
>> >
>> > I'm also glad to hear any other opinions!
>> >
>> > Thanks,
>> > Wei
>> >
>> >
>> >> 在 2019年9月26日，15:23，Dian Fu <[hidden email] <mailto:[hidden email]>> 写道：
>> >>
>> >> Hi Wei,
>> >>
>> >> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
>> >>
>> >> Thanks,
>> >> Dian
>> >>
>> >>> 在 2019年9月26日，下午12:23，jincheng sun <[hidden email] <mailto:[hidden email]>> 写道：
>> >>>
>> >>> Thanks for bring up the discussion, Wei.
>> >>> Overall the design doc looks good. I have left a few comments.
>> >>>
>> >>> BTW: Dependency Management is very important for Python UDFs, welcome
>> >>> anyone left your suggestions!
>> >>>
>> >>> Best,
>> >>> Jincheng
>> >>>
>> >>> Wei Zhong <[hidden email] <mailto:[hidden email]>> 于2019年9月26日周四 上午11:59写道：
>> >>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>> >>>> python UDF, the environment and dependency management of users' python code
>> >>>> has not been fully discussed.
>> >>>>
>> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
>> >>>> Dependency Management". Here is the design doc I drafted:
>> >>>>
>> >>>>
>> >>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing <https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing>
>> >>>>
>> >>>> Please take a look, and feedbacks are welcome.
>> >>>>
>> >>>> Thanks,
>> >>>> Wei
>> >>>>
>> >>>> [1]:
>> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table>
>> >>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>>
>> >>>>
>> >>>>
>> >>
>> >
>>
>