[DISCUSS] Move docker development into versioned branches

[Chesnay Schepler-3]

Hello everyone,

Currently, all development in docker-flink occurs on the master branch,
for all releases at once. The master branch also serves as a publishing
area for our docker files.

This means that all versions share the same generators, templates (i.e.,
production code that we plugin in some variables like the Flink version)
and test setup.

So far this worked fine because all versions worked the same, but we are
now coming to a point where we are adding version-specific features,
like support for Java 11 (FLINK-16260) and per-job-clusters (FLINK-17164).

By virtue of all files being shared this means that we would have to
introduce version checks in both the generators, templates and test
code, so things continue to work for all versions. This is rather
painful to do, and presents maintenance problems as these conditions
must of course be updated at some point, and diverging behaviors require
all versions to be tested on each PR (in contrast to the current state
where we only test 1 version).

It is also simply different to the process we're using for
flink/flink-shaded, creating an additional entry barrier.

I propose moving the development of the docker files into dedicated
branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
opened against dev-master, possibly ported to other versions, and when
it is time for a release we will update the master branch with the
latest dockerfiles. Quite similar to how things work in the Flink repo.

The master branch would continue to be our release publishing area to
stick to docker conventions.

Regards,

Chesnay

[David Anderson-3]

Makes sense to me. I think this would align well enough with user
expectations, and be more straightforward.

David

On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler <[hidden email]> wrote:

> Hello everyone,
>
> Currently, all development in docker-flink occurs on the master branch,
> for all releases at once. The master branch also serves as a publishing
> area for our docker files.
>
> This means that all versions share the same generators, templates (i.e.,
> production code that we plugin in some variables like the Flink version)
> and test setup.
>
> So far this worked fine because all versions worked the same, but we are
> now coming to a point where we are adding version-specific features,
> like support for Java 11 (FLINK-16260) and per-job-clusters (FLINK-17164).
>
> By virtue of all files being shared this means that we would have to
> introduce version checks in both the generators, templates and test
> code, so things continue to work for all versions. This is rather
> painful to do, and presents maintenance problems as these conditions
> must of course be updated at some point, and diverging behaviors require
> all versions to be tested on each PR (in contrast to the current state
> where we only test 1 version).
>
> It is also simply different to the process we're using for
> flink/flink-shaded, creating an additional entry barrier.
>
> I propose moving the development of the docker files into dedicated
> branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> opened against dev-master, possibly ported to other versions, and when
> it is time for a release we will update the master branch with the
> latest dockerfiles. Quite similar to how things work in the Flink repo.
>
> The master branch would continue to be our release publishing area to
> stick to docker conventions.
>
> Regards,
>
> Chesnay
>
>

[Till Rohrmann]

Thanks for starting this discussion Chesnay. Your proposal sounds good to
me. I can see how the current setup makes the development of version
specific features impractical. Hence, +1 for the proposed changes.

Cheers,
Till

On Mon, Apr 27, 2020 at 12:19 PM David Anderson <[hidden email]>
wrote:

> Makes sense to me. I think this would align well enough with user
> expectations, and be more straightforward.
>
> David
>
> On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler <[hidden email]>
> wrote:
>
> > Hello everyone,
> >
> > Currently, all development in docker-flink occurs on the master branch,
> > for all releases at once. The master branch also serves as a publishing
> > area for our docker files.
> >
> > This means that all versions share the same generators, templates (i.e.,
> > production code that we plugin in some variables like the Flink version)
> > and test setup.
> >
> > So far this worked fine because all versions worked the same, but we are
> > now coming to a point where we are adding version-specific features,
> > like support for Java 11 (FLINK-16260) and per-job-clusters
> (FLINK-17164).
> >
> > By virtue of all files being shared this means that we would have to
> > introduce version checks in both the generators, templates and test
> > code, so things continue to work for all versions. This is rather
> > painful to do, and presents maintenance problems as these conditions
> > must of course be updated at some point, and diverging behaviors require
> > all versions to be tested on each PR (in contrast to the current state
> > where we only test 1 version).
> >
> > It is also simply different to the process we're using for
> > flink/flink-shaded, creating an additional entry barrier.
> >
> > I propose moving the development of the docker files into dedicated
> > branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> > opened against dev-master, possibly ported to other versions, and when
> > it is time for a release we will update the master branch with the
> > latest dockerfiles. Quite similar to how things work in the Flink repo.
> >
> > The master branch would continue to be our release publishing area to
> > stick to docker conventions.
> >
> > Regards,
> >
> > Chesnay
> >
> >
>

[Patrick Lucas-2]

This also sounds good to me. The master branch will continue to be the
publishing area for the release Dockerfiles and we can support changes
between Flink versions without additional complexity.

My only comment would be that I hope we can still keep eventual convergence
in mind—it should be possible, with some changes in Flink, to make the
differences in the Docker packaging between versions negligible, to the
point that this split might stop being necessary.

On Mon, Apr 27, 2020 at 3:07 PM Till Rohrmann <[hidden email]> wrote:

> Thanks for starting this discussion Chesnay. Your proposal sounds good to
> me. I can see how the current setup makes the development of version
> specific features impractical. Hence, +1 for the proposed changes.
>
> Cheers,
> Till
>
> On Mon, Apr 27, 2020 at 12:19 PM David Anderson <[hidden email]>
> wrote:
>
> > Makes sense to me. I think this would align well enough with user
> > expectations, and be more straightforward.
> >
> > David
> >
> > On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler <[hidden email]>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Currently, all development in docker-flink occurs on the master branch,
> > > for all releases at once. The master branch also serves as a publishing
> > > area for our docker files.
> > >
> > > This means that all versions share the same generators, templates
> (i.e.,
> > > production code that we plugin in some variables like the Flink
> version)
> > > and test setup.
> > >
> > > So far this worked fine because all versions worked the same, but we
> are
> > > now coming to a point where we are adding version-specific features,
> > > like support for Java 11 (FLINK-16260) and per-job-clusters
> > (FLINK-17164).
> > >
> > > By virtue of all files being shared this means that we would have to
> > > introduce version checks in both the generators, templates and test
> > > code, so things continue to work for all versions. This is rather
> > > painful to do, and presents maintenance problems as these conditions
> > > must of course be updated at some point, and diverging behaviors
> require
> > > all versions to be tested on each PR (in contrast to the current state
> > > where we only test 1 version).
> > >
> > > It is also simply different to the process we're using for
> > > flink/flink-shaded, creating an additional entry barrier.
> > >
> > > I propose moving the development of the docker files into dedicated
> > > branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be
> > > opened against dev-master, possibly ported to other versions, and when
> > > it is time for a release we will update the master branch with the
> > > latest dockerfiles. Quite similar to how things work in the Flink repo.
> > >
> > > The master branch would continue to be our release publishing area to
> > > stick to docker conventions.
> > >
> > > Regards,
> > >
> > > Chesnay
> > >
> > >
> >
>