[DISCUSS] How to prepare the Python environment of PyFlink release in the current Flink release process

[Dian Fu-2]

Hi all,

We have reached a consensus that the PyFlink package should be published to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has already been created and available to use now [2]. It means that we could publish PyFlink to PyPI in the coming releases and it also means that additional steps will be added to the normal process of the Flink release to prepare the PyFlink release package.

It needs a proper Python environment(i.e. Python 3.5+, setuptools, etc) to build the PyFlink package. There are two options in my mind to prepare the Python environment:
1) Reuse the script lint-python.sh defined in flink-python module to create the required virtual environment and build the PyFlink package using the created virtual environment.
2) It's assumed that the local Python environment is properly installed and ready to use. The Python environment requirement will be documented at the page "Create a Flink Release" and validation check could also be added in create_binary_release.sh to throw an meaningful error with hints how to fix it if it's not correct.

Option 1:
Pros:
- It's transparent for release managers.
Cons:
- It needs to prepare the virtual environment during preparing the PyFlink release package and it will take some several minutes as it need to download a few binaries.

Option 2:
Pros:
- There is no need to prepare the virtual environment if the local environment is already properly configured.
Cons:
- It requires the release managers to prepare the local Python environment and not all the people are familiar with Python and it's a burden for release managers.

Personally I prefer to option 1).

Looking forward to your feedback!

PS: I think this issue could also be discussed in the JIRA. But I tend to bring up the discussion to ML as it introduces an additional step to the release process and I think this should be visible to the community and it should be well discussed. Besides, we could also get more feedback.

Regards,
Dian

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
[2] https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307

[Jeff Zhang]

I am a little confused, why we need to prepare python environment in
release. Shouldn't that be done when user start to use pyflink ?
Or do you mean to set up python environment for pyflink's CI build ?

Regarding this problem  "It needs a proper Python environment(i.e. Python
3.5+, setuptools, etc) to build the PyFlink package"
Would the build fail if I use python 3.4 ?


Dian Fu <[hidden email]> 于2019年10月29日周二 上午11:01写道：

> Hi all,
>
> We have reached a consensus that the PyFlink package should be published
> to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has already
> been created and available to use now [2]. It means that we could publish
> PyFlink to PyPI in the coming releases and it also means that additional
> steps will be added to the normal process of the Flink release to prepare
> the PyFlink release package.
>
> It needs a proper Python environment(i.e. Python 3.5+, setuptools, etc) to
> build the PyFlink package. There are two options in my mind to prepare the
> Python environment:
> 1) Reuse the script lint-python.sh defined in flink-python module to
> create the required virtual environment and build the PyFlink package using
> the created virtual environment.
> 2) It's assumed that the local Python environment is properly installed
> and ready to use. The Python environment requirement will be documented at
> the page "Create a Flink Release" and validation check could also be added
> in create_binary_release.sh to throw an meaningful error with hints how to
> fix it if it's not correct.
>
> Option 1:
> Pros:
> - It's transparent for release managers.
> Cons:
> - It needs to prepare the virtual environment during preparing the PyFlink
> release package and it will take some several minutes as it need to
> download a few binaries.
>
> Option 2:
> Pros:
> - There is no need to prepare the virtual environment if the local
> environment is already properly configured.
> Cons:
> - It requires the release managers to prepare the local Python environment
> and not all the people are familiar with Python and it's a burden for
> release managers.
>
> Personally I prefer to option 1).
>
> Looking forward to your feedback!
>
> PS: I think this issue could also be discussed in the JIRA. But I tend to
> bring up the discussion to ML as it introduces an additional step to the
> release process and I think this should be visible to the community and it
> should be well discussed. Besides, we could also get more feedback.
>
> Regards,
> Dian
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
> [2]
> https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307

--
Best Regards

Jeff Zhang

[jincheng sun]

Hi,
Thanks for bringing up the discussion Dian.
+1 for the #1.

Hi Jeff, this changes is for the PyFlink release, i.e.,The release manager
should build the release package for Pyflink, and prepare the python
environment during the building. Since 1.10 we only support python 3.5+, so
it will throw an exception if you use python 3.4.

Best,
Jincheng


Jeff Zhang <[hidden email]> 于2019年10月29日周二 上午11:55写道：

> I am a little confused, why we need to prepare python environment in
> release. Shouldn't that be done when user start to use pyflink ?
> Or do you mean to set up python environment for pyflink's CI build ?
>
> Regarding this problem  "It needs a proper Python environment(i.e. Python
> 3.5+, setuptools, etc) to build the PyFlink package"
> Would the build fail if I use python 3.4 ?
>
>
> Dian Fu <[hidden email]> 于2019年10月29日周二 上午11:01写道：
>
> > Hi all,
> >
> > We have reached a consensus that the PyFlink package should be published
> > to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has already
> > been created and available to use now [2]. It means that we could publish
> > PyFlink to PyPI in the coming releases and it also means that additional
> > steps will be added to the normal process of the Flink release to prepare
> > the PyFlink release package.
> >
> > It needs a proper Python environment(i.e. Python 3.5+, setuptools, etc)
> to
> > build the PyFlink package. There are two options in my mind to prepare
> the
> > Python environment:
> > 1) Reuse the script lint-python.sh defined in flink-python module to
> > create the required virtual environment and build the PyFlink package
> using
> > the created virtual environment.
> > 2) It's assumed that the local Python environment is properly installed
> > and ready to use. The Python environment requirement will be documented
> at
> > the page "Create a Flink Release" and validation check could also be
> added
> > in create_binary_release.sh to throw an meaningful error with hints how
> to
> > fix it if it's not correct.
> >
> > Option 1:
> > Pros:
> > - It's transparent for release managers.
> > Cons:
> > - It needs to prepare the virtual environment during preparing the
> PyFlink
> > release package and it will take some several minutes as it need to
> > download a few binaries.
> >
> > Option 2:
> > Pros:
> > - There is no need to prepare the virtual environment if the local
> > environment is already properly configured.
> > Cons:
> > - It requires the release managers to prepare the local Python
> environment
> > and not all the people are familiar with Python and it's a burden for
> > release managers.
> >
> > Personally I prefer to option 1).
> >
> > Looking forward to your feedback!
> >
> > PS: I think this issue could also be discussed in the JIRA. But I tend to
> > bring up the discussion to ML as it introduces an additional step to the
> > release process and I think this should be visible to the community and
> it
> > should be well discussed. Besides, we could also get more feedback.
> >
> > Regards,
> > Dian
> >
> > [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
> > [2]
> >
> https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

[Till Rohrmann]

Thanks for bringing this topic up Dian. I'd be in favour of option #1
because this would also allow to create reproducible builds.

Cheers,
Till

On Tue, Oct 29, 2019 at 5:28 AM jincheng sun <[hidden email]>
wrote:

> Hi,
> Thanks for bringing up the discussion Dian.
> +1 for the #1.
>
> Hi Jeff, this changes is for the PyFlink release, i.e.,The release manager
> should build the release package for Pyflink, and prepare the python
> environment during the building. Since 1.10 we only support python 3.5+, so
> it will throw an exception if you use python 3.4.
>
> Best,
> Jincheng
>
>
> Jeff Zhang <[hidden email]> 于2019年10月29日周二 上午11:55写道：
>
> > I am a little confused, why we need to prepare python environment in
> > release. Shouldn't that be done when user start to use pyflink ?
> > Or do you mean to set up python environment for pyflink's CI build ?
> >
> > Regarding this problem  "It needs a proper Python environment(i.e. Python
> > 3.5+, setuptools, etc) to build the PyFlink package"
> > Would the build fail if I use python 3.4 ?
> >
> >
> > Dian Fu <[hidden email]> 于2019年10月29日周二 上午11:01写道：
> >
> > > Hi all,
> > >
> > > We have reached a consensus that the PyFlink package should be
> published
> > > to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has
> already
> > > been created and available to use now [2]. It means that we could
> publish
> > > PyFlink to PyPI in the coming releases and it also means that
> additional
> > > steps will be added to the normal process of the Flink release to
> prepare
> > > the PyFlink release package.
> > >
> > > It needs a proper Python environment(i.e. Python 3.5+, setuptools, etc)
> > to
> > > build the PyFlink package. There are two options in my mind to prepare
> > the
> > > Python environment:
> > > 1) Reuse the script lint-python.sh defined in flink-python module to
> > > create the required virtual environment and build the PyFlink package
> > using
> > > the created virtual environment.
> > > 2) It's assumed that the local Python environment is properly installed
> > > and ready to use. The Python environment requirement will be documented
> > at
> > > the page "Create a Flink Release" and validation check could also be
> > added
> > > in create_binary_release.sh to throw an meaningful error with hints how
> > to
> > > fix it if it's not correct.
> > >
> > > Option 1:
> > > Pros:
> > > - It's transparent for release managers.
> > > Cons:
> > > - It needs to prepare the virtual environment during preparing the
> > PyFlink
> > > release package and it will take some several minutes as it need to
> > > download a few binaries.
> > >
> > > Option 2:
> > > Pros:
> > > - There is no need to prepare the virtual environment if the local
> > > environment is already properly configured.
> > > Cons:
> > > - It requires the release managers to prepare the local Python
> > environment
> > > and not all the people are familiar with Python and it's a burden for
> > > release managers.
> > >
> > > Personally I prefer to option 1).
> > >
> > > Looking forward to your feedback!
> > >
> > > PS: I think this issue could also be discussed in the JIRA. But I tend
> to
> > > bring up the discussion to ML as it introduces an additional step to
> the
> > > release process and I think this should be visible to the community and
> > it
> > > should be well discussed. Besides, we could also get more feedback.
> > >
> > > Regards,
> > > Dian
> > >
> > > [1]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
> > > [2]
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

[Hequn Cheng]

Hi Dian,

Thanks a lot for bringing the discussion.
It would be a headache to address these environmental problems, so +1 for
option #1 to use the virtual environment.

Best, Hequn

On Tue, Oct 29, 2019 at 9:31 PM Till Rohrmann <[hidden email]> wrote:

> Thanks for bringing this topic up Dian. I'd be in favour of option #1
> because this would also allow to create reproducible builds.
>
> Cheers,
> Till
>
> On Tue, Oct 29, 2019 at 5:28 AM jincheng sun <[hidden email]>
> wrote:
>
> > Hi,
> > Thanks for bringing up the discussion Dian.
> > +1 for the #1.
> >
> > Hi Jeff, this changes is for the PyFlink release, i.e.,The release
> manager
> > should build the release package for Pyflink, and prepare the python
> > environment during the building. Since 1.10 we only support python 3.5+,
> so
> > it will throw an exception if you use python 3.4.
> >
> > Best,
> > Jincheng
> >
> >
> > Jeff Zhang <[hidden email]> 于2019年10月29日周二 上午11:55写道：
> >
> > > I am a little confused, why we need to prepare python environment in
> > > release. Shouldn't that be done when user start to use pyflink ?
> > > Or do you mean to set up python environment for pyflink's CI build ?
> > >
> > > Regarding this problem  "It needs a proper Python environment(i.e.
> Python
> > > 3.5+, setuptools, etc) to build the PyFlink package"
> > > Would the build fail if I use python 3.4 ?
> > >
> > >
> > > Dian Fu <[hidden email]> 于2019年10月29日周二 上午11:01写道：
> > >
> > > > Hi all,
> > > >
> > > > We have reached a consensus that the PyFlink package should be
> > published
> > > > to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has
> > already
> > > > been created and available to use now [2]. It means that we could
> > publish
> > > > PyFlink to PyPI in the coming releases and it also means that
> > additional
> > > > steps will be added to the normal process of the Flink release to
> > prepare
> > > > the PyFlink release package.
> > > >
> > > > It needs a proper Python environment(i.e. Python 3.5+, setuptools,
> etc)
> > > to
> > > > build the PyFlink package. There are two options in my mind to
> prepare
> > > the
> > > > Python environment:
> > > > 1) Reuse the script lint-python.sh defined in flink-python module to
> > > > create the required virtual environment and build the PyFlink package
> > > using
> > > > the created virtual environment.
> > > > 2) It's assumed that the local Python environment is properly
> installed
> > > > and ready to use. The Python environment requirement will be
> documented
> > > at
> > > > the page "Create a Flink Release" and validation check could also be
> > > added
> > > > in create_binary_release.sh to throw an meaningful error with hints
> how
> > > to
> > > > fix it if it's not correct.
> > > >
> > > > Option 1:
> > > > Pros:
> > > > - It's transparent for release managers.
> > > > Cons:
> > > > - It needs to prepare the virtual environment during preparing the
> > > PyFlink
> > > > release package and it will take some several minutes as it need to
> > > > download a few binaries.
> > > >
> > > > Option 2:
> > > > Pros:
> > > > - There is no need to prepare the virtual environment if the local
> > > > environment is already properly configured.
> > > > Cons:
> > > > - It requires the release managers to prepare the local Python
> > > environment
> > > > and not all the people are familiar with Python and it's a burden for
> > > > release managers.
> > > >
> > > > Personally I prefer to option 1).
> > > >
> > > > Looking forward to your feedback!
> > > >
> > > > PS: I think this issue could also be discussed in the JIRA. But I
> tend
> > to
> > > > bring up the discussion to ML as it introduces an additional step to
> > the
> > > > release process and I think this should be visible to the community
> and
> > > it
> > > > should be well discussed. Besides, we could also get more feedback.
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > [1]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
> > > > [2]
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
>

[Dian Fu-2]

Hi all,

Thanks a lot for the discussion. As all the people tend to option #1, we will take option#1 as the solution for this issue. Thanks again!

Thanks,
Dian

> 在 2019年10月30日，上午9:58，Hequn Cheng <[hidden email]> 写道：
>
> Hi Dian,
>
> Thanks a lot for bringing the discussion.
> It would be a headache to address these environmental problems, so +1 for
> option #1 to use the virtual environment.
>
> Best, Hequn
>
> On Tue, Oct 29, 2019 at 9:31 PM Till Rohrmann <[hidden email]> wrote:
>
>> Thanks for bringing this topic up Dian. I'd be in favour of option #1
>> because this would also allow to create reproducible builds.
>>
>> Cheers,
>> Till
>>
>> On Tue, Oct 29, 2019 at 5:28 AM jincheng sun <[hidden email]>
>> wrote:
>>
>>> Hi,
>>> Thanks for bringing up the discussion Dian.
>>> +1 for the #1.
>>>
>>> Hi Jeff, this changes is for the PyFlink release, i.e.,The release
>> manager
>>> should build the release package for Pyflink, and prepare the python
>>> environment during the building. Since 1.10 we only support python 3.5+,
>> so
>>> it will throw an exception if you use python 3.4.
>>>
>>> Best,
>>> Jincheng
>>>
>>>
>>> Jeff Zhang <[hidden email]> 于2019年10月29日周二 上午11:55写道：
>>>
>>>> I am a little confused, why we need to prepare python environment in
>>>> release. Shouldn't that be done when user start to use pyflink ?
>>>> Or do you mean to set up python environment for pyflink's CI build ?
>>>>
>>>> Regarding this problem  "It needs a proper Python environment(i.e.
>> Python
>>>> 3.5+, setuptools, etc) to build the PyFlink package"
>>>> Would the build fail if I use python 3.4 ?
>>>>
>>>>
>>>> Dian Fu <[hidden email]> 于2019年10月29日周二 上午11:01写道：
>>>>
>>>>> Hi all,
>>>>>
>>>>> We have reached a consensus that the PyFlink package should be
>>> published
>>>>> to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has
>>> already
>>>>> been created and available to use now [2]. It means that we could
>>> publish
>>>>> PyFlink to PyPI in the coming releases and it also means that
>>> additional
>>>>> steps will be added to the normal process of the Flink release to
>>> prepare
>>>>> the PyFlink release package.
>>>>>
>>>>> It needs a proper Python environment(i.e. Python 3.5+, setuptools,
>> etc)
>>>> to
>>>>> build the PyFlink package. There are two options in my mind to
>> prepare
>>>> the
>>>>> Python environment:
>>>>> 1) Reuse the script lint-python.sh defined in flink-python module to
>>>>> create the required virtual environment and build the PyFlink package
>>>> using
>>>>> the created virtual environment.
>>>>> 2) It's assumed that the local Python environment is properly
>> installed
>>>>> and ready to use. The Python environment requirement will be
>> documented
>>>> at
>>>>> the page "Create a Flink Release" and validation check could also be
>>>> added
>>>>> in create_binary_release.sh to throw an meaningful error with hints
>> how
>>>> to
>>>>> fix it if it's not correct.
>>>>>
>>>>> Option 1:
>>>>> Pros:
>>>>> - It's transparent for release managers.
>>>>> Cons:
>>>>> - It needs to prepare the virtual environment during preparing the
>>>> PyFlink
>>>>> release package and it will take some several minutes as it need to
>>>>> download a few binaries.
>>>>>
>>>>> Option 2:
>>>>> Pros:
>>>>> - There is no need to prepare the virtual environment if the local
>>>>> environment is already properly configured.
>>>>> Cons:
>>>>> - It requires the release managers to prepare the local Python
>>>> environment
>>>>> and not all the people are familiar with Python and it's a burden for
>>>>> release managers.
>>>>>
>>>>> Personally I prefer to option 1).
>>>>>
>>>>> Looking forward to your feedback!
>>>>>
>>>>> PS: I think this issue could also be discussed in the JIRA. But I
>> tend
>>> to
>>>>> bring up the discussion to ML as it introduces an additional step to
>>> the
>>>>> release process and I think this should be visible to the community
>> and
>>>> it
>>>>> should be well discussed. Besides, we could also get more feedback.
>>>>>
>>>>> Regards,
>>>>> Dian
>>>>>
>>>>> [1]
>>>>>
>>>>
>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
>>>>> [2]
>>>>>
>>>>
>>>
>> https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>>
>>