[DISCUSS] Intermediary releases of the flink-docker images

[Ismaël Mejía]

Hello,

I wanted to discuss about a subject that was shortly mentioned during the docker
unification thread (and in some other PR) that is the release of docker images
independent of major Flink releases.

In the past when the docker images were maintained outside of the Apache
repository we usually did intermediary releases to fix issues or to add new
functionalities that were independent of the Flink releases and specific of the
docker images.

There are two major cases when this happened:

1. If the upstream official docker images maintainers requested us to do the
   changes or there was some breakage because some upstream change (this is rare
   but has happened in the past).

2. If we wanted to fix issues or add new functionality to the images that was
   independent of the full Flink release.

We have been working on having Java 11 based images available and this is an
example of (2), where we want to publish these images based on the already
released 1.10.0 version.

So I would like to know your opinion on how should we proceed in the future.
Ideally we should wait for the major release, but the reality is that (1) can
happen and (2) can benefit end users.

So what should we do? Can we do these updates without a formal release as we did
before, or does it make sense to follow a release process with the corresponding
vote for the docker images? or are there other alternatives?

Regards,
Ismaël

[Tang Jinxin]

I am confused why docker upstream can influence inner process seriously,may be the jvm thread?In my opinion,docker is just a child space of os. xiaoxingstack 邮箱：[hidden email] 签名由 网易邮箱大师 定制 On 04/22/2020 20:55, Ismaël Mejía wrote: Hello, I wanted to discuss about a subject that was shortly mentioned during the docker unification thread (and in some other PR) that is the release of docker images independent of major Flink releases. In the past when the docker images were maintained outside of the Apache repository we usually did intermediary releases to fix issues or to add new functionalities that were independent of the Flink releases and specific of the docker images. There are two major cases when this happened: 1. If the upstream official docker images maintainers requested us to do the   changes or there was some breakage because some upstream change (this is rare   but has happened in the past). 2. If we wanted to fix issues or add new functionality to the images that was   independent of the full Flink release. We have been working on having Java 11 based images available and this is an example of (2), where we want to publish these images based on the already released 1.10.0 version. So I would like to know your opinion on how should we proceed in the future. Ideally we should wait for the major release, but the reality is that (1) can happen and (2) can benefit end users. So what should we do? Can we do these updates without a formal release as we did before, or does it make sense to follow a release process with the corresponding vote for the docker images? or are there other alternatives? Regards, Ismaël

[Tang Jinxin]

maybe could try someway like foreachpartition in foreachrdd,which will not together to driver take too extra consumption. xiaoxingstack 邮箱：[hidden email] 签名由 网易邮箱大师 定制 On 04/22/2020 20:55, Ismaël Mejía wrote: Hello, I wanted to discuss about a subject that was shortly mentioned during the docker unification thread (and in some other PR) that is the release of docker images independent of major Flink releases. In the past when the docker images were maintained outside of the Apache repository we usually did intermediary releases to fix issues or to add new functionalities that were independent of the Flink releases and specific of the docker images. There are two major cases when this happened: 1. If the upstream official docker images maintainers requested us to do the   changes or there was some breakage because some upstream change (this is rare   but has happened in the past). 2. If we wanted to fix issues or add new functionality to the images that was   independent of the full Flink release. We have been working on having Java 11 based images available and this is an example of (2), where we want to publish these images based on the already released 1.10.0 version. So I would like to know your opinion on how should we proceed in the future. Ideally we should wait for the major release, but the reality is that (1) can happen and (2) can benefit end users. So what should we do? Can we do these updates without a formal release as we did before, or does it make sense to follow a release process with the corresponding vote for the docker images? or are there other alternatives? Regards, Ismaël

[Chesnay Schepler-3]

We can create additional releases independent of Flink, but they will
have to go through a formal release process in any case.

On 22/04/2020 14:55, Ismaël Mejía wrote:

> Hello,
>
> I wanted to discuss about a subject that was shortly mentioned during the docker
> unification thread (and in some other PR) that is the release of docker images
> independent of major Flink releases.
>
> In the past when the docker images were maintained outside of the Apache
> repository we usually did intermediary releases to fix issues or to add new
> functionalities that were independent of the Flink releases and specific of the
> docker images.
>
> There are two major cases when this happened:
>
> 1. If the upstream official docker images maintainers requested us to do the
>     changes or there was some breakage because some upstream change (this is rare
>     but has happened in the past).
>
> 2. If we wanted to fix issues or add new functionality to the images that was
>     independent of the full Flink release.
>
> We have been working on having Java 11 based images available and this is an
> example of (2), where we want to publish these images based on the already
> released 1.10.0 version.
>
> So I would like to know your opinion on how should we proceed in the future.
> Ideally we should wait for the major release, but the reality is that (1) can
> happen and (2) can benefit end users.
>
> So what should we do? Can we do these updates without a formal release as we did
> before, or does it make sense to follow a release process with the corresponding
> vote for the docker images? or are there other alternatives?
>
> Regards,
> Ismaël
>

[Robert Metzger]

Thanks for starting the thread!

I would consider the docker images of Flink convenience binary releases
that can happen any time. I believe a simplified, but formal release
process would be appropriate (preview / staging images for the community to
validate & vote, then release to docker hub).

On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]> wrote:

> We can create additional releases independent of Flink, but they will
> have to go through a formal release process in any case.
>
> On 22/04/2020 14:55, Ismaël Mejía wrote:
> > Hello,
> >
> > I wanted to discuss about a subject that was shortly mentioned during
> the docker
> > unification thread (and in some other PR) that is the release of docker
> images
> > independent of major Flink releases.
> >
> > In the past when the docker images were maintained outside of the Apache
> > repository we usually did intermediary releases to fix issues or to add
> new
> > functionalities that were independent of the Flink releases and specific
> of the
> > docker images.
> >
> > There are two major cases when this happened:
> >
> > 1. If the upstream official docker images maintainers requested us to do
> the
> >     changes or there was some breakage because some upstream change
> (this is rare
> >     but has happened in the past).
> >
> > 2. If we wanted to fix issues or add new functionality to the images
> that was
> >     independent of the full Flink release.
> >
> > We have been working on having Java 11 based images available and this
> is an
> > example of (2), where we want to publish these images based on the
> already
> > released 1.10.0 version.
> >
> > So I would like to know your opinion on how should we proceed in the
> future.
> > Ideally we should wait for the major release, but the reality is that
> (1) can
> > happen and (2) can benefit end users.
> >
> > So what should we do? Can we do these updates without a formal release
> as we did
> > before, or does it make sense to follow a release process with the
> corresponding
> > vote for the docker images? or are there other alternatives?
> >
> > Regards,
> > Ismaël
> >
>
>

[Niels Basjes]

Hi,

In my opinion the docker images are essentially simply differently packed
binary releases.

This becomes more true when in the future deploying a Flink application to
kubernetes simply pulls the correct binary from a docker hub. Because of
these kinds of use cases I disagree with Robert that these are just
convenience releases.

To me this means that the docker images should be released at the same time
the other artifacts are released. This also includes shapshot releases. So
the build of the docker images should be an integral part of the build.

Do note that there will be multiple sub-versions for each release with
variations in for example the used Scala version and/or JDK. All published
at the same moment.

Niels Basjes

On Mon, Apr 27, 2020, 10:26 Robert Metzger <[hidden email]> wrote:

> Thanks for starting the thread!
>
> I would consider the docker images of Flink convenience binary releases
> that can happen any time. I believe a simplified, but formal release
> process would be appropriate (preview / staging images for the community to
> validate & vote, then release to docker hub).
>
> On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]>
> wrote:
>
> > We can create additional releases independent of Flink, but they will
> > have to go through a formal release process in any case.
> >
> > On 22/04/2020 14:55, Ismaël Mejía wrote:
> > > Hello,
> > >
> > > I wanted to discuss about a subject that was shortly mentioned during
> > the docker
> > > unification thread (and in some other PR) that is the release of docker
> > images
> > > independent of major Flink releases.
> > >
> > > In the past when the docker images were maintained outside of the
> Apache
> > > repository we usually did intermediary releases to fix issues or to add
> > new
> > > functionalities that were independent of the Flink releases and
> specific
> > of the
> > > docker images.
> > >
> > > There are two major cases when this happened:
> > >
> > > 1. If the upstream official docker images maintainers requested us to
> do
> > the
> > >     changes or there was some breakage because some upstream change
> > (this is rare
> > >     but has happened in the past).
> > >
> > > 2. If we wanted to fix issues or add new functionality to the images
> > that was
> > >     independent of the full Flink release.
> > >
> > > We have been working on having Java 11 based images available and this
> > is an
> > > example of (2), where we want to publish these images based on the
> > already
> > > released 1.10.0 version.
> > >
> > > So I would like to know your opinion on how should we proceed in the
> > future.
> > > Ideally we should wait for the major release, but the reality is that
> > (1) can
> > > happen and (2) can benefit end users.
> > >
> > > So what should we do? Can we do these updates without a formal release
> > as we did
> > > before, or does it make sense to follow a release process with the
> > corresponding
> > > vote for the docker images? or are there other alternatives?
> > >
> > > Regards,
> > > Ismaël
> > >
> >
> >
>

[Aljoscha Krettek-2]

Hi Niels,

I think Robert was referring to the fact that Apache considers only the
source release to be "the release", everything else is called
convenience release.

Best,
Aljoscha

On 27.04.20 19:43, Niels Basjes wrote:

> Hi,
>
> In my opinion the docker images are essentially simply differently packed
> binary releases.
>
> This becomes more true when in the future deploying a Flink application to
> kubernetes simply pulls the correct binary from a docker hub. Because of
> these kinds of use cases I disagree with Robert that these are just
> convenience releases.
>
> To me this means that the docker images should be released at the same time
> the other artifacts are released. This also includes shapshot releases. So
> the build of the docker images should be an integral part of the build.
>
> Do note that there will be multiple sub-versions for each release with
> variations in for example the used Scala version and/or JDK. All published
> at the same moment.
>
> Niels Basjes
>
> On Mon, Apr 27, 2020, 10:26 Robert Metzger <[hidden email]> wrote:
>
>> Thanks for starting the thread!
>>
>> I would consider the docker images of Flink convenience binary releases
>> that can happen any time. I believe a simplified, but formal release
>> process would be appropriate (preview / staging images for the community to
>> validate & vote, then release to docker hub).
>>
>> On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]>
>> wrote:
>>
>>> We can create additional releases independent of Flink, but they will
>>> have to go through a formal release process in any case.
>>>
>>> On 22/04/2020 14:55, Ismaël Mejía wrote:
>>>> Hello,
>>>>
>>>> I wanted to discuss about a subject that was shortly mentioned during
>>> the docker
>>>> unification thread (and in some other PR) that is the release of docker
>>> images
>>>> independent of major Flink releases.
>>>>
>>>> In the past when the docker images were maintained outside of the
>> Apache
>>>> repository we usually did intermediary releases to fix issues or to add
>>> new
>>>> functionalities that were independent of the Flink releases and
>> specific
>>> of the
>>>> docker images.
>>>>
>>>> There are two major cases when this happened:
>>>>
>>>> 1. If the upstream official docker images maintainers requested us to
>> do
>>> the
>>>>      changes or there was some breakage because some upstream change
>>> (this is rare
>>>>      but has happened in the past).
>>>>
>>>> 2. If we wanted to fix issues or add new functionality to the images
>>> that was
>>>>      independent of the full Flink release.
>>>>
>>>> We have been working on having Java 11 based images available and this
>>> is an
>>>> example of (2), where we want to publish these images based on the
>>> already
>>>> released 1.10.0 version.
>>>>
>>>> So I would like to know your opinion on how should we proceed in the
>>> future.
>>>> Ideally we should wait for the major release, but the reality is that
>>> (1) can
>>>> happen and (2) can benefit end users.
>>>>
>>>> So what should we do? Can we do these updates without a formal release
>>> as we did
>>>> before, or does it make sense to follow a release process with the
>>> corresponding
>>>> vote for the docker images? or are there other alternatives?
>>>>
>>>> Regards,
>>>> Ismaël
>>>>
>>>
>>>
>>
>

[Chesnay Schepler-3]

I agree with Niels that if a Flink release is made it should be
accompanied by the Dockerfiles for that version, and release them at once.
This makes sense for testing purposes alone, and I view Docker as just
another distribution channel like Maven central.
The reverse isn't necessarily true though, as in we can release an
updated version of the Dockerfiles without an accompanying Flink release.
This is just for convenience so we can fix issues on the Docker side faster.
This is a fair compromise imo.

I do wonder though whether the Dockerfiles really are binary releases.
Aren't they more of an instruction set to assemble one?
Given that we actively point other parties to files&commits in our repo
(which I suppose by definition is the source), then it seems
questionable to categorize this as a binary release.

On 28/04/2020 09:30, Aljoscha Krettek wrote:

> Hi Niels,
>
> I think Robert was referring to the fact that Apache considers only
> the source release to be "the release", everything else is called
> convenience release.
>
> Best,
> Aljoscha
>
> On 27.04.20 19:43, Niels Basjes wrote:
>> Hi,
>>
>> In my opinion the docker images are essentially simply differently
>> packed
>> binary releases.
>>
>> This becomes more true when in the future deploying a Flink
>> application to
>> kubernetes simply pulls the correct binary from a docker hub. Because of
>> these kinds of use cases I disagree with Robert that these are just
>> convenience releases.
>>
>> To me this means that the docker images should be released at the
>> same time
>> the other artifacts are released. This also includes shapshot
>> releases. So
>> the build of the docker images should be an integral part of the build.
>>
>> Do note that there will be multiple sub-versions for each release with
>> variations in for example the used Scala version and/or JDK. All
>> published
>> at the same moment.
>>
>> Niels Basjes
>>
>> On Mon, Apr 27, 2020, 10:26 Robert Metzger <[hidden email]> wrote:
>>
>>> Thanks for starting the thread!
>>>
>>> I would consider the docker images of Flink convenience binary releases
>>> that can happen any time. I believe a simplified, but formal release
>>> process would be appropriate (preview / staging images for the
>>> community to
>>> validate & vote, then release to docker hub).
>>>
>>> On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]>
>>> wrote:
>>>
>>>> We can create additional releases independent of Flink, but they will
>>>> have to go through a formal release process in any case.
>>>>
>>>> On 22/04/2020 14:55, Ismaël Mejía wrote:
>>>>> Hello,
>>>>>
>>>>> I wanted to discuss about a subject that was shortly mentioned during
>>>> the docker
>>>>> unification thread (and in some other PR) that is the release of
>>>>> docker
>>>> images
>>>>> independent of major Flink releases.
>>>>>
>>>>> In the past when the docker images were maintained outside of the
>>> Apache
>>>>> repository we usually did intermediary releases to fix issues or
>>>>> to add
>>>> new
>>>>> functionalities that were independent of the Flink releases and
>>> specific
>>>> of the
>>>>> docker images.
>>>>>
>>>>> There are two major cases when this happened:
>>>>>
>>>>> 1. If the upstream official docker images maintainers requested us to
>>> do
>>>> the
>>>>>      changes or there was some breakage because some upstream change
>>>> (this is rare
>>>>>      but has happened in the past).
>>>>>
>>>>> 2. If we wanted to fix issues or add new functionality to the images
>>>> that was
>>>>>      independent of the full Flink release.
>>>>>
>>>>> We have been working on having Java 11 based images available and
>>>>> this
>>>> is an
>>>>> example of (2), where we want to publish these images based on the
>>>> already
>>>>> released 1.10.0 version.
>>>>>
>>>>> So I would like to know your opinion on how should we proceed in the
>>>> future.
>>>>> Ideally we should wait for the major release, but the reality is that
>>>> (1) can
>>>>> happen and (2) can benefit end users.
>>>>>
>>>>> So what should we do? Can we do these updates without a formal
>>>>> release
>>>> as we did
>>>>> before, or does it make sense to follow a release process with the
>>>> corresponding
>>>>> vote for the docker images? or are there other alternatives?
>>>>>
>>>>> Regards,
>>>>> Ismaël
>>>>>
>>>>
>>>>
>>>
>>
>
>

[Ismaël Mejía]

> To me this means that the docker images should be released at the same time the other artifacts are released. This also includes shapshot releases. So the build of the docker images should be an integral part of the build.

This is already the case since the last release, what this thread is
about are the intermediary releases that correspond to an upstream
release. Let’s call those the 1.10.0-dockerN. Let’s not forget that
our released docker images are not fully immutable artifacts because
upstream docker official images maintainers may (and regularly do)
update them for example to fix security issues on the parent images or
to catch new versions of Java or the base OS in our case openjdk and
debian.

These releases may be required to fix issues or include new features
that are independent of Flink releases (we can also ignore and be
tight to the Flink release cycle to simplify the process), but
remember that the upstream docker images maintainers may ‘force us’ to
do updates (this rarely happens but when it happens is something we
cannot ignore).

> I do wonder though whether the Dockerfiles really are binary releases.

The Dockerfiles are not binary releases but are the base of the docker
image releases let’s not forget that since we want the images to be
official docker-images release
https://github.com/docker-library/official-images we have to align
with their processes too.

I suppose the wisest is to align with the ASF vote/release process and
that this produces a new PR of and image in the official-images repo
(which would be the dockee equivalent of publishing to central). There
are probably some minor issues that we might find in the way, so far
the only one I can think of is how would we identify intermediary
docker releases but apart of this I am +1 on ‘formal’ release process
even if I found it too heavy we are an ASF project so we should align
with its rules.

Also this goes inline with the existing proposal by Chesnay on “Moving
docker development into versioned branches”
https://lists.apache.org/thread.html/rbb2569d24b35a40adb01b7abfe62fc1ada0408539403ef826491d0ee%40%3Cdev.flink.apache.org%3E

Any other opinions?

If we reach consensus maybe we should do an initial extra release and
VOTE to test the process. I would like to volunteer to do this and
document the process (we could for example do this to include the Java
11 version), but well let’s wait for other opinions first.

Ismaël

ps.  I should probably be accompanied by a committer at some steps
because I could lack some permissions to do the release.

On Tue, Apr 28, 2020 at 1:17 PM Chesnay Schepler <[hidden email]> wrote:

>
> I agree with Niels that if a Flink release is made it should be
> accompanied by the Dockerfiles for that version, and release them at once.
> This makes sense for testing purposes alone, and I view Docker as just
> another distribution channel like Maven central.
> The reverse isn't necessarily true though, as in we can release an
> updated version of the Dockerfiles without an accompanying Flink release.
> This is just for convenience so we can fix issues on the Docker side faster.
> This is a fair compromise imo.
>
> I do wonder though whether the Dockerfiles really are binary releases.
> Aren't they more of an instruction set to assemble one?
> Given that we actively point other parties to files&commits in our repo
> (which I suppose by definition is the source), then it seems
> questionable to categorize this as a binary release.
>
> On 28/04/2020 09:30, Aljoscha Krettek wrote:
> > Hi Niels,
> >
> > I think Robert was referring to the fact that Apache considers only
> > the source release to be "the release", everything else is called
> > convenience release.
> >
> > Best,
> > Aljoscha
> >
> > On 27.04.20 19:43, Niels Basjes wrote:
> >> Hi,
> >>
> >> In my opinion the docker images are essentially simply differently
> >> packed
> >> binary releases.
> >>
> >> This becomes more true when in the future deploying a Flink
> >> application to
> >> kubernetes simply pulls the correct binary from a docker hub. Because of
> >> these kinds of use cases I disagree with Robert that these are just
> >> convenience releases.
> >>
> >> To me this means that the docker images should be released at the
> >> same time
> >> the other artifacts are released. This also includes shapshot
> >> releases. So
> >> the build of the docker images should be an integral part of the build.
> >>
> >> Do note that there will be multiple sub-versions for each release with
> >> variations in for example the used Scala version and/or JDK. All
> >> published
> >> at the same moment.
> >>
> >> Niels Basjes
> >>
> >> On Mon, Apr 27, 2020, 10:26 Robert Metzger <[hidden email]> wrote:
> >>
> >>> Thanks for starting the thread!
> >>>
> >>> I would consider the docker images of Flink convenience binary releases
> >>> that can happen any time. I believe a simplified, but formal release
> >>> process would be appropriate (preview / staging images for the
> >>> community to
> >>> validate & vote, then release to docker hub).
> >>>
> >>> On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]>
> >>> wrote:
> >>>
> >>>> We can create additional releases independent of Flink, but they will
> >>>> have to go through a formal release process in any case.
> >>>>
> >>>> On 22/04/2020 14:55, Ismaël Mejía wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I wanted to discuss about a subject that was shortly mentioned during
> >>>> the docker
> >>>>> unification thread (and in some other PR) that is the release of
> >>>>> docker
> >>>> images
> >>>>> independent of major Flink releases.
> >>>>>
> >>>>> In the past when the docker images were maintained outside of the
> >>> Apache
> >>>>> repository we usually did intermediary releases to fix issues or
> >>>>> to add
> >>>> new
> >>>>> functionalities that were independent of the Flink releases and
> >>> specific
> >>>> of the
> >>>>> docker images.
> >>>>>
> >>>>> There are two major cases when this happened:
> >>>>>
> >>>>> 1. If the upstream official docker images maintainers requested us to
> >>> do
> >>>> the
> >>>>>      changes or there was some breakage because some upstream change
> >>>> (this is rare
> >>>>>      but has happened in the past).
> >>>>>
> >>>>> 2. If we wanted to fix issues or add new functionality to the images
> >>>> that was
> >>>>>      independent of the full Flink release.
> >>>>>
> >>>>> We have been working on having Java 11 based images available and
> >>>>> this
> >>>> is an
> >>>>> example of (2), where we want to publish these images based on the
> >>>> already
> >>>>> released 1.10.0 version.
> >>>>>
> >>>>> So I would like to know your opinion on how should we proceed in the
> >>>> future.
> >>>>> Ideally we should wait for the major release, but the reality is that
> >>>> (1) can
> >>>>> happen and (2) can benefit end users.
> >>>>>
> >>>>> So what should we do? Can we do these updates without a formal
> >>>>> release
> >>>> as we did
> >>>>> before, or does it make sense to follow a release process with the
> >>>> corresponding
> >>>>> vote for the docker images? or are there other alternatives?
> >>>>>
> >>>>> Regards,
> >>>>> Ismaël
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >
> >
>

[Yang Wang]

Thanks for starting this discussion.

In general, i am also in favor of making docker image release could be
partly decoupled from Flink release. However, i think it should only happen
when we want to make some changes that are independent from Flink
majar release(e.g. JDK11, change base image, install more tools, upgrade the
docker entrypoint, etc.). If the Flink release branch codes have changed, we
should not build them in a new docker image. Instead, if we could provide
the daily updating snapshot for docker image, it will be very helpful for
the users
who want to have a taste of new features or benefit from urgent hotfix.


Best,
Yang

Ismaël Mejía <[hidden email]> 于2020年4月28日周二 下午11:07写道：

> > To me this means that the docker images should be released at the same
> time the other artifacts are released. This also includes shapshot
> releases. So the build of the docker images should be an integral part of
> the build.
>
> This is already the case since the last release, what this thread is
> about are the intermediary releases that correspond to an upstream
> release. Let’s call those the 1.10.0-dockerN. Let’s not forget that
> our released docker images are not fully immutable artifacts because
> upstream docker official images maintainers may (and regularly do)
> update them for example to fix security issues on the parent images or
> to catch new versions of Java or the base OS in our case openjdk and
> debian.
>
> These releases may be required to fix issues or include new features
> that are independent of Flink releases (we can also ignore and be
> tight to the Flink release cycle to simplify the process), but
> remember that the upstream docker images maintainers may ‘force us’ to
> do updates (this rarely happens but when it happens is something we
> cannot ignore).
>
> > I do wonder though whether the Dockerfiles really are binary releases.
>
> The Dockerfiles are not binary releases but are the base of the docker
> image releases let’s not forget that since we want the images to be
> official docker-images release
> https://github.com/docker-library/official-images we have to align
> with their processes too.
>
> I suppose the wisest is to align with the ASF vote/release process and
> that this produces a new PR of and image in the official-images repo
> (which would be the dockee equivalent of publishing to central). There
> are probably some minor issues that we might find in the way, so far
> the only one I can think of is how would we identify intermediary
> docker releases but apart of this I am +1 on ‘formal’ release process
> even if I found it too heavy we are an ASF project so we should align
> with its rules.
>
> Also this goes inline with the existing proposal by Chesnay on “Moving
> docker development into versioned branches”
>
> https://lists.apache.org/thread.html/rbb2569d24b35a40adb01b7abfe62fc1ada0408539403ef826491d0ee%40%3Cdev.flink.apache.org%3E
>
> Any other opinions?
>
> If we reach consensus maybe we should do an initial extra release and
> VOTE to test the process. I would like to volunteer to do this and
> document the process (we could for example do this to include the Java
> 11 version), but well let’s wait for other opinions first.
>
> Ismaël
>
> ps.  I should probably be accompanied by a committer at some steps
> because I could lack some permissions to do the release.
>
> On Tue, Apr 28, 2020 at 1:17 PM Chesnay Schepler <[hidden email]>
> wrote:
> >
> > I agree with Niels that if a Flink release is made it should be
> > accompanied by the Dockerfiles for that version, and release them at
> once.
> > This makes sense for testing purposes alone, and I view Docker as just
> > another distribution channel like Maven central.
> > The reverse isn't necessarily true though, as in we can release an
> > updated version of the Dockerfiles without an accompanying Flink release.
> > This is just for convenience so we can fix issues on the Docker side
> faster.
> > This is a fair compromise imo.
> >
> > I do wonder though whether the Dockerfiles really are binary releases.
> > Aren't they more of an instruction set to assemble one?
> > Given that we actively point other parties to files&commits in our repo
> > (which I suppose by definition is the source), then it seems
> > questionable to categorize this as a binary release.
> >
> > On 28/04/2020 09:30, Aljoscha Krettek wrote:
> > > Hi Niels,
> > >
> > > I think Robert was referring to the fact that Apache considers only
> > > the source release to be "the release", everything else is called
> > > convenience release.
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 27.04.20 19:43, Niels Basjes wrote:
> > >> Hi,
> > >>
> > >> In my opinion the docker images are essentially simply differently
> > >> packed
> > >> binary releases.
> > >>
> > >> This becomes more true when in the future deploying a Flink
> > >> application to
> > >> kubernetes simply pulls the correct binary from a docker hub. Because
> of
> > >> these kinds of use cases I disagree with Robert that these are just
> > >> convenience releases.
> > >>
> > >> To me this means that the docker images should be released at the
> > >> same time
> > >> the other artifacts are released. This also includes shapshot
> > >> releases. So
> > >> the build of the docker images should be an integral part of the
> build.
> > >>
> > >> Do note that there will be multiple sub-versions for each release with
> > >> variations in for example the used Scala version and/or JDK. All
> > >> published
> > >> at the same moment.
> > >>
> > >> Niels Basjes
> > >>
> > >> On Mon, Apr 27, 2020, 10:26 Robert Metzger <[hidden email]>
> wrote:
> > >>
> > >>> Thanks for starting the thread!
> > >>>
> > >>> I would consider the docker images of Flink convenience binary
> releases
> > >>> that can happen any time. I believe a simplified, but formal release
> > >>> process would be appropriate (preview / staging images for the
> > >>> community to
> > >>> validate & vote, then release to docker hub).
> > >>>
> > >>> On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]
> >
> > >>> wrote:
> > >>>
> > >>>> We can create additional releases independent of Flink, but they
> will
> > >>>> have to go through a formal release process in any case.
> > >>>>
> > >>>> On 22/04/2020 14:55, Ismaël Mejía wrote:
> > >>>>> Hello,
> > >>>>>
> > >>>>> I wanted to discuss about a subject that was shortly mentioned
> during
> > >>>> the docker
> > >>>>> unification thread (and in some other PR) that is the release of
> > >>>>> docker
> > >>>> images
> > >>>>> independent of major Flink releases.
> > >>>>>
> > >>>>> In the past when the docker images were maintained outside of the
> > >>> Apache
> > >>>>> repository we usually did intermediary releases to fix issues or
> > >>>>> to add
> > >>>> new
> > >>>>> functionalities that were independent of the Flink releases and
> > >>> specific
> > >>>> of the
> > >>>>> docker images.
> > >>>>>
> > >>>>> There are two major cases when this happened:
> > >>>>>
> > >>>>> 1. If the upstream official docker images maintainers requested us
> to
> > >>> do
> > >>>> the
> > >>>>>      changes or there was some breakage because some upstream
> change
> > >>>> (this is rare
> > >>>>>      but has happened in the past).
> > >>>>>
> > >>>>> 2. If we wanted to fix issues or add new functionality to the
> images
> > >>>> that was
> > >>>>>      independent of the full Flink release.
> > >>>>>
> > >>>>> We have been working on having Java 11 based images available and
> > >>>>> this
> > >>>> is an
> > >>>>> example of (2), where we want to publish these images based on the
> > >>>> already
> > >>>>> released 1.10.0 version.
> > >>>>>
> > >>>>> So I would like to know your opinion on how should we proceed in
> the
> > >>>> future.
> > >>>>> Ideally we should wait for the major release, but the reality is
> that
> > >>>> (1) can
> > >>>>> happen and (2) can benefit end users.
> > >>>>>
> > >>>>> So what should we do? Can we do these updates without a formal
> > >>>>> release
> > >>>> as we did
> > >>>>> before, or does it make sense to follow a release process with the
> > >>>> corresponding
> > >>>>> vote for the docker images? or are there other alternatives?
> > >>>>>
> > >>>>> Regards,
> > >>>>> Ismaël
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> > >
> >
>

[Chesnay Schepler-3]

We will not advertise snapshot artifacts to users. This is simply not
allowed by ASF rules.

On 29/04/2020 04:08, Yang Wang wrote:

> Thanks for starting this discussion.
>
> In general, i am also in favor of making docker image release could be
> partly decoupled from Flink release. However, i think it should only happen
> when we want to make some changes that are independent from Flink
> majar release(e.g. JDK11, change base image, install more tools, upgrade the
> docker entrypoint, etc.). If the Flink release branch codes have changed, we
> should not build them in a new docker image. Instead, if we could provide
> the daily updating snapshot for docker image, it will be very helpful for
> the users
> who want to have a taste of new features or benefit from urgent hotfix.
>
>
> Best,
> Yang
>
> Ismaël Mejía <[hidden email]> 于2020年4月28日周二 下午11:07写道：
>
>>> To me this means that the docker images should be released at the same
>> time the other artifacts are released. This also includes shapshot
>> releases. So the build of the docker images should be an integral part of
>> the build.
>>
>> This is already the case since the last release, what this thread is
>> about are the intermediary releases that correspond to an upstream
>> release. Let’s call those the 1.10.0-dockerN. Let’s not forget that
>> our released docker images are not fully immutable artifacts because
>> upstream docker official images maintainers may (and regularly do)
>> update them for example to fix security issues on the parent images or
>> to catch new versions of Java or the base OS in our case openjdk and
>> debian.
>>
>> These releases may be required to fix issues or include new features
>> that are independent of Flink releases (we can also ignore and be
>> tight to the Flink release cycle to simplify the process), but
>> remember that the upstream docker images maintainers may ‘force us’ to
>> do updates (this rarely happens but when it happens is something we
>> cannot ignore).
>>
>>> I do wonder though whether the Dockerfiles really are binary releases.
>> The Dockerfiles are not binary releases but are the base of the docker
>> image releases let’s not forget that since we want the images to be
>> official docker-images release
>> https://github.com/docker-library/official-images we have to align
>> with their processes too.
>>
>> I suppose the wisest is to align with the ASF vote/release process and
>> that this produces a new PR of and image in the official-images repo
>> (which would be the dockee equivalent of publishing to central). There
>> are probably some minor issues that we might find in the way, so far
>> the only one I can think of is how would we identify intermediary
>> docker releases but apart of this I am +1 on ‘formal’ release process
>> even if I found it too heavy we are an ASF project so we should align
>> with its rules.
>>
>> Also this goes inline with the existing proposal by Chesnay on “Moving
>> docker development into versioned branches”
>>
>> https://lists.apache.org/thread.html/rbb2569d24b35a40adb01b7abfe62fc1ada0408539403ef826491d0ee%40%3Cdev.flink.apache.org%3E
>>
>> Any other opinions?
>>
>> If we reach consensus maybe we should do an initial extra release and
>> VOTE to test the process. I would like to volunteer to do this and
>> document the process (we could for example do this to include the Java
>> 11 version), but well let’s wait for other opinions first.
>>
>> Ismaël
>>
>> ps.  I should probably be accompanied by a committer at some steps
>> because I could lack some permissions to do the release.
>>
>> On Tue, Apr 28, 2020 at 1:17 PM Chesnay Schepler <[hidden email]>
>> wrote:
>>> I agree with Niels that if a Flink release is made it should be
>>> accompanied by the Dockerfiles for that version, and release them at
>> once.
>>> This makes sense for testing purposes alone, and I view Docker as just
>>> another distribution channel like Maven central.
>>> The reverse isn't necessarily true though, as in we can release an
>>> updated version of the Dockerfiles without an accompanying Flink release.
>>> This is just for convenience so we can fix issues on the Docker side
>> faster.
>>> This is a fair compromise imo.
>>>
>>> I do wonder though whether the Dockerfiles really are binary releases.
>>> Aren't they more of an instruction set to assemble one?
>>> Given that we actively point other parties to files&commits in our repo
>>> (which I suppose by definition is the source), then it seems
>>> questionable to categorize this as a binary release.
>>>
>>> On 28/04/2020 09:30, Aljoscha Krettek wrote:
>>>> Hi Niels,
>>>>
>>>> I think Robert was referring to the fact that Apache considers only
>>>> the source release to be "the release", everything else is called
>>>> convenience release.
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>> On 27.04.20 19:43, Niels Basjes wrote:
>>>>> Hi,
>>>>>
>>>>> In my opinion the docker images are essentially simply differently
>>>>> packed
>>>>> binary releases.
>>>>>
>>>>> This becomes more true when in the future deploying a Flink
>>>>> application to
>>>>> kubernetes simply pulls the correct binary from a docker hub. Because
>> of
>>>>> these kinds of use cases I disagree with Robert that these are just
>>>>> convenience releases.
>>>>>
>>>>> To me this means that the docker images should be released at the
>>>>> same time
>>>>> the other artifacts are released. This also includes shapshot
>>>>> releases. So
>>>>> the build of the docker images should be an integral part of the
>> build.
>>>>> Do note that there will be multiple sub-versions for each release with
>>>>> variations in for example the used Scala version and/or JDK. All
>>>>> published
>>>>> at the same moment.
>>>>>
>>>>> Niels Basjes
>>>>>
>>>>> On Mon, Apr 27, 2020, 10:26 Robert Metzger <[hidden email]>
>> wrote:
>>>>>> Thanks for starting the thread!
>>>>>>
>>>>>> I would consider the docker images of Flink convenience binary
>> releases
>>>>>> that can happen any time. I believe a simplified, but formal release
>>>>>> process would be appropriate (preview / staging images for the
>>>>>> community to
>>>>>> validate & vote, then release to docker hub).
>>>>>>
>>>>>> On Wed, Apr 22, 2020 at 3:29 PM Chesnay Schepler <[hidden email]
>>>>>> wrote:
>>>>>>
>>>>>>> We can create additional releases independent of Flink, but they
>> will
>>>>>>> have to go through a formal release process in any case.
>>>>>>>
>>>>>>> On 22/04/2020 14:55, Ismaël Mejía wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I wanted to discuss about a subject that was shortly mentioned
>> during
>>>>>>> the docker
>>>>>>>> unification thread (and in some other PR) that is the release of
>>>>>>>> docker
>>>>>>> images
>>>>>>>> independent of major Flink releases.
>>>>>>>>
>>>>>>>> In the past when the docker images were maintained outside of the
>>>>>> Apache
>>>>>>>> repository we usually did intermediary releases to fix issues or
>>>>>>>> to add
>>>>>>> new
>>>>>>>> functionalities that were independent of the Flink releases and
>>>>>> specific
>>>>>>> of the
>>>>>>>> docker images.
>>>>>>>>
>>>>>>>> There are two major cases when this happened:
>>>>>>>>
>>>>>>>> 1. If the upstream official docker images maintainers requested us
>> to
>>>>>> do
>>>>>>> the
>>>>>>>>       changes or there was some breakage because some upstream
>> change
>>>>>>> (this is rare
>>>>>>>>       but has happened in the past).
>>>>>>>>
>>>>>>>> 2. If we wanted to fix issues or add new functionality to the
>> images
>>>>>>> that was
>>>>>>>>       independent of the full Flink release.
>>>>>>>>
>>>>>>>> We have been working on having Java 11 based images available and
>>>>>>>> this
>>>>>>> is an
>>>>>>>> example of (2), where we want to publish these images based on the
>>>>>>> already
>>>>>>>> released 1.10.0 version.
>>>>>>>>
>>>>>>>> So I would like to know your opinion on how should we proceed in
>> the
>>>>>>> future.
>>>>>>>> Ideally we should wait for the major release, but the reality is
>> that
>>>>>>> (1) can
>>>>>>>> happen and (2) can benefit end users.
>>>>>>>>
>>>>>>>> So what should we do? Can we do these updates without a formal
>>>>>>>> release
>>>>>>> as we did
>>>>>>>> before, or does it make sense to follow a release process with the
>>>>>>> corresponding
>>>>>>>> vote for the docker images? or are there other alternatives?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Ismaël
>>>>>>>>
>>>>>>>
>>>>