[DISCUSS] Leverage Maven Wrapper

[tison]

Hi devs,

Flink forces a fixed version(3.2.5) of Maven while higher version suffers
from shade issues and
so on.

Since different projects have different requirement of Maven. It seems a
good idea we add a
maven wrapper[1] in our repository which reduces our developers burden.

Any thoughts?

Best,
tison.

[1] https://github.com/takari/maven-wrapper

[Chesnay Schepler-3]

You don't necessarily need to use maven 3.2.5, you just have to know
what to watch out for.
For that reason we are _not_ forcing maven 3.2.5 in general, but for
releases only to be on the safe side.

Some time ago Gradle was brought up as a potential replacement for
maven, and I'd like to see that discussion concluded before making other
major changes to the maven development process.

Overall I'm sympathetic to the idea, but not really sold.
If we switch to gradle we obviously don't need it (duh);
if we stick with maven we will have to find a solution for the 3.2.5
limitation eventually, and I'd much rather solve this problem than keep
working around the limitation.

On 03/06/2020 15:48, tison wrote:

> Hi devs,
>
> Flink forces a fixed version(3.2.5) of Maven while higher version suffers
> from shade issues and
> so on.
>
> Since different projects have different requirement of Maven. It seems a
> good idea we add a
> maven wrapper[1] in our repository which reduces our developers burden.
>
> Any thoughts?
>
> Best,
> tison.
>
> [1] https://github.com/takari/maven-wrapper
>

[Robert Metzger]

I just reached out to the users@maven mailing list again to check if
there's any resolution for shading behavior post 3.2.5 [1]


[1]
https://lists.apache.org/thread.html/8b2dcf462de814d06d8e30bafce2c886217c5790a3ee07d33d0b8dfc%40%3Cusers.maven.apache.org%3E

On Thu, Jun 4, 2020 at 3:08 PM Chesnay Schepler <[hidden email]> wrote:

> You don't necessarily need to use maven 3.2.5, you just have to know
> what to watch out for.
> For that reason we are _not_ forcing maven 3.2.5 in general, but for
> releases only to be on the safe side.
>
> Some time ago Gradle was brought up as a potential replacement for
> maven, and I'd like to see that discussion concluded before making other
> major changes to the maven development process.
>
> Overall I'm sympathetic to the idea, but not really sold.
> If we switch to gradle we obviously don't need it (duh);
> if we stick with maven we will have to find a solution for the 3.2.5
> limitation eventually, and I'd much rather solve this problem than keep
> working around the limitation.
>
> On 03/06/2020 15:48, tison wrote:
> > Hi devs,
> >
> > Flink forces a fixed version(3.2.5) of Maven while higher version suffers
> > from shade issues and
> > so on.
> >
> > Since different projects have different requirement of Maven. It seems a
> > good idea we add a
> > maven wrapper[1] in our repository which reduces our developers burden.
> >
> > Any thoughts?
> >
> > Best,
> > tison.
> >
> > [1] https://github.com/takari/maven-wrapper
> >
>
>

[Chesnay Schepler-3]

It's not that /difficult /for us to work around it mind you; we "just"
have to

a) separate the distribution packaging from the flink-dist jar from the
distribution assembly, so we can have dependencies on the various
opt/plugins modules without pulling in dependencies into the flink-dist jar,
b) get rid of as many intermediate shading instances as possible, (i.e.,
flink-runtime which is bundled into flink-dist)
c) in cases where we cannot get rid of intermediate shading (e.g.,
because we want to have 2 different versions of a single dependency
bundled with separate packages), add exclusions to the shaded
dependencies. (this is essentially what the shade-plugin does for the
dependency-reduced pom)

On 03/07/2020 10:12, Robert Metzger wrote:

> I just reached out to the users@maven mailing list again to check if
> there's any resolution for shading behavior post 3.2.5 [1]
>
>
> [1]
> https://lists.apache.org/thread.html/8b2dcf462de814d06d8e30bafce2c886217c5790a3ee07d33d0b8dfc%40%3Cusers.maven.apache.org%3E
>
> On Thu, Jun 4, 2020 at 3:08 PM Chesnay Schepler <[hidden email]> wrote:
>
>> You don't necessarily need to use maven 3.2.5, you just have to know
>> what to watch out for.
>> For that reason we are _not_ forcing maven 3.2.5 in general, but for
>> releases only to be on the safe side.
>>
>> Some time ago Gradle was brought up as a potential replacement for
>> maven, and I'd like to see that discussion concluded before making other
>> major changes to the maven development process.
>>
>> Overall I'm sympathetic to the idea, but not really sold.
>> If we switch to gradle we obviously don't need it (duh);
>> if we stick with maven we will have to find a solution for the 3.2.5
>> limitation eventually, and I'd much rather solve this problem than keep
>> working around the limitation.
>>
>> On 03/06/2020 15:48, tison wrote:
>>> Hi devs,
>>>
>>> Flink forces a fixed version(3.2.5) of Maven while higher version suffers
>>> from shade issues and
>>> so on.
>>>
>>> Since different projects have different requirement of Maven. It seems a
>>> good idea we add a
>>> maven wrapper[1] in our repository which reduces our developers burden.
>>>
>>> Any thoughts?
>>>
>>> Best,
>>> tison.
>>>
>>> [1] https://github.com/takari/maven-wrapper
>>>
>>

[Robert Metzger]

We could also build releases by calling "mvn package" again in
"flink-dist". But all these solutions are far from elegant.

Ideally the Maven folks have implemented something nicer by now. Let's see
what they say.

On Fri, Jul 3, 2020 at 1:20 PM Chesnay Schepler <[hidden email]> wrote:

> It's not that *difficult *for us to work around it mind you; we "just"
> have to
>
> a) separate the distribution packaging from the flink-dist jar from the
> distribution assembly, so we can have dependencies on the various
> opt/plugins modules without pulling in dependencies into the flink-dist jar,
> b) get rid of as many intermediate shading instances as possible, (i.e.,
> flink-runtime which is bundled into flink-dist)
> c) in cases where we cannot get rid of intermediate shading (e.g., because
> we want to have 2 different versions of a single dependency bundled with
> separate packages), add exclusions to the shaded dependencies. (this is
> essentially what the shade-plugin does for the dependency-reduced pom)
>
> On 03/07/2020 10:12, Robert Metzger wrote:
>
> I just reached out to the users@maven mailing list again to check if
> there's any resolution for shading behavior post 3.2.5 [1]
>
>
> [1]https://lists.apache.org/thread.html/8b2dcf462de814d06d8e30bafce2c886217c5790a3ee07d33d0b8dfc%40%3Cusers.maven.apache.org%3E
>
> On Thu, Jun 4, 2020 at 3:08 PM Chesnay Schepler <[hidden email]> <[hidden email]> wrote:
>
>
> You don't necessarily need to use maven 3.2.5, you just have to know
> what to watch out for.
> For that reason we are _not_ forcing maven 3.2.5 in general, but for
> releases only to be on the safe side.
>
> Some time ago Gradle was brought up as a potential replacement for
> maven, and I'd like to see that discussion concluded before making other
> major changes to the maven development process.
>
> Overall I'm sympathetic to the idea, but not really sold.
> If we switch to gradle we obviously don't need it (duh);
> if we stick with maven we will have to find a solution for the 3.2.5
> limitation eventually, and I'd much rather solve this problem than keep
> working around the limitation.
>
> On 03/06/2020 15:48, tison wrote:
>
> Hi devs,
>
> Flink forces a fixed version(3.2.5) of Maven while higher version suffers
> from shade issues and
> so on.
>
> Since different projects have different requirement of Maven. It seems a
> good idea we add a
> maven wrapper[1] in our repository which reduces our developers burden.
>
> Any thoughts?
>
> Best,
> tison.
>
> [1] https://github.com/takari/maven-wrapper
>
>
>

[Chesnay Schepler-3]

That's incorrect; you need to do the same for filesystems, as one
example. So, build everything -> build filesystems -> build flink-dist.

On 03/07/2020 14:31, Robert Metzger wrote:

> We could also build releases by calling "mvn package" again in
> "flink-dist". But all these solutions are far from elegant.
>
> Ideally the Maven folks have implemented something nicer by now. Let's
> see what they say.
>
> On Fri, Jul 3, 2020 at 1:20 PM Chesnay Schepler <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     It's not that /difficult /for us to work around it mind you; we
>     "just" have to
>
>     a) separate the distribution packaging from the flink-dist jar
>     from the distribution assembly, so we can have dependencies on the
>     various opt/plugins modules without pulling in dependencies into
>     the flink-dist jar,
>     b) get rid of as many intermediate shading instances as possible,
>     (i.e., flink-runtime which is bundled into flink-dist)
>     c) in cases where we cannot get rid of intermediate shading (e.g.,
>     because we want to have 2 different versions of a single
>     dependency bundled with separate packages), add exclusions to the
>     shaded dependencies. (this is essentially what the shade-plugin
>     does for the dependency-reduced pom)
>
>     On 03/07/2020 10:12, Robert Metzger wrote:
>>     I just reached out to the users@maven mailing list again to check if
>>     there's any resolution for shading behavior post 3.2.5 [1]
>>
>>
>>     [1]
>>     https://lists.apache.org/thread.html/8b2dcf462de814d06d8e30bafce2c886217c5790a3ee07d33d0b8dfc%40%3Cusers.maven.apache.org%3E
>>
>>     On Thu, Jun 4, 2020 at 3:08 PM Chesnay Schepler<[hidden email]>  <mailto:[hidden email]>  wrote:
>>
>>>     You don't necessarily need to use maven 3.2.5, you just have to know
>>>     what to watch out for.
>>>     For that reason we are _not_ forcing maven 3.2.5 in general, but for
>>>     releases only to be on the safe side.
>>>
>>>     Some time ago Gradle was brought up as a potential replacement for
>>>     maven, and I'd like to see that discussion concluded before making other
>>>     major changes to the maven development process.
>>>
>>>     Overall I'm sympathetic to the idea, but not really sold.
>>>     If we switch to gradle we obviously don't need it (duh);
>>>     if we stick with maven we will have to find a solution for the 3.2.5
>>>     limitation eventually, and I'd much rather solve this problem than keep
>>>     working around the limitation.
>>>
>>>     On 03/06/2020 15:48, tison wrote:
>>>>     Hi devs,
>>>>
>>>>     Flink forces a fixed version(3.2.5) of Maven while higher version suffers
>>>>     from shade issues and
>>>>     so on.
>>>>
>>>>     Since different projects have different requirement of Maven. It seems a
>>>>     good idea we add a
>>>>     maven wrapper[1] in our repository which reduces our developers burden.
>>>>
>>>>     Any thoughts?
>>>>
>>>>     Best,
>>>>     tison.
>>>>
>>>>     [1]https://github.com/takari/maven-wrapper
>>>>
>