Quickstart POMs

[Ufuk Celebi-2]

Hey all,

looking at the quickstarts for the upcoming release, I am a little confused:

We use the shade plugin to build a fat jar and exclude some of Flink's dependencies (actual and transitive). Then we have a build profile (build-jar), which sets the Flink dependencies to provided.

1. Is there a difference between these two approaches? @Robert and Stephan: can you comment on this please? I would like to fix this for the release. The current quickstart pom looks somewhat unclean.

From my understanding both approaches are equivalent and actually the "provided" variant w/o cumbersome shade plugin exclusion configuration should be preferred, e.g.

<dependency>
   <groupId>org.apache.flink</groupId>
   <artifactId>flink-java</artifactId>
   <version>0.9.0</version>
   <scope>provided</scope>
</dependency>

... (other Flink deps for the quickstart)

+ simple shade plugin configuration to build a fat jar (don't need to set any exclusions (which is doomed to get out-of-sync)).


If I am not overlooking anything, this approach is much cleaner and should be the default.

– Ufuk

[Fabian Hueske-2]

I don't think that many users care about the internals of the quickstart
pom file and are just happy if it works.

I would not cancel the ongoing vote for that.
We might fix it if the vote is canceled, but it has no priority, IMO.

2015-06-18 15:32 GMT+02:00 Ufuk Celebi <[hidden email]>:

> Hey all,
>
> looking at the quickstarts for the upcoming release, I am a little
> confused:
>
> We use the shade plugin to build a fat jar and exclude some of Flink's
> dependencies (actual and transitive). Then we have a build profile
> (build-jar), which sets the Flink dependencies to provided.
>
> 1. Is there a difference between these two approaches? @Robert and
> Stephan: can you comment on this please? I would like to fix this for the
> release. The current quickstart pom looks somewhat unclean.
>
> From my understanding both approaches are equivalent and actually the
> "provided" variant w/o cumbersome shade plugin exclusion configuration
> should be preferred, e.g.
>
> <dependency>
>    <groupId>org.apache.flink</groupId>
>    <artifactId>flink-java</artifactId>
>    <version>0.9.0</version>
>    <scope>provided</scope>
> </dependency>
>
> ... (other Flink deps for the quickstart)
>
> + simple shade plugin configuration to build a fat jar (don't need to set
> any exclusions (which is doomed to get out-of-sync)).
>
>
> If I am not overlooking anything, this approach is much cleaner and should
> be the default.
>
> – Ufuk
>
>
>

[Ufuk Celebi-2]

On 18 Jun 2015, at 16:49, Fabian Hueske <[hidden email]> wrote:

> I don't think that many users care about the internals of the quickstart
> pom file and are just happy if it works.

I agree with this. That's exactly why I think it's better to keep it clean; with the current pom you actually do have to worry about the internals. ;)

[Fabian Hueske-2]

Why?

mvn package

builds the program correctly, no?
On Jun 18, 2015 16:53, "Ufuk Celebi" <[hidden email]> wrote:

>
> On 18 Jun 2015, at 16:49, Fabian Hueske <[hidden email]> wrote:
>
> > I don't think that many users care about the internals of the quickstart
> > pom file and are just happy if it works.
>
> I agree with this. That's exactly why I think it's better to keep it
> clean; with the current pom you actually do have to worry about the
> internals. ;)

[Ufuk Celebi-2]

On 18 Jun 2015, at 16:58, Fabian Hueske <[hidden email]> wrote:

> Why?
>
> mvn package
>
> builds the program correctly, no?

Yes, but:

- Dependencies not specified by the user may be included (Metrics, javaassist)
- Dependencies specified by the user may be excluded
- If you use the build-jar profile you have to understand what the difference to the default profile is and then you have to include your dependencies again for the profile
- The pom comments are confusing

[Aljoscha Krettek-2]

I'm also for simplification but let's hear what those who put the build-jar
profile there have to say about it.?

On Thu, 18 Jun 2015 at 17:25 Ufuk Celebi <[hidden email]> wrote:

>
> On 18 Jun 2015, at 16:58, Fabian Hueske <[hidden email]> wrote:
>
> > Why?
> >
> > mvn package
> >
> > builds the program correctly, no?
>
> Yes, but:
>
> - Dependencies not specified by the user may be included (Metrics,
> javaassist)
> - Dependencies specified by the user may be excluded
> - If you use the build-jar profile you have to understand what the
> difference to the default profile is and then you have to include your
> dependencies again for the profile
> - The pom comments are confusing

[Chiwan Park]

Is it okay when the user runs the built jar in LocalEnvironment? (Just run with `java -jar` command.)
I know that it is special case but it is a possible scenario for local testing.

If we change Quickstart POM to use provided scope for dependencies, we should add a guide about this into document.

Regards,
Chiwan Park

> On Jun 19, 2015, at 12:53 AM, Aljoscha Krettek <[hidden email]> wrote:
>
> I'm also for simplification but let's hear what those who put the build-jar
> profile there have to say about it.?
>
> On Thu, 18 Jun 2015 at 17:25 Ufuk Celebi <[hidden email]> wrote:
>
>>
>> On 18 Jun 2015, at 16:58, Fabian Hueske <[hidden email]> wrote:
>>
>>> Why?
>>>
>>> mvn package
>>>
>>> builds the program correctly, no?
>>
>> Yes, but:
>>
>> - Dependencies not specified by the user may be included (Metrics,
>> javaassist)
>> - Dependencies specified by the user may be excluded
>> - If you use the build-jar profile you have to understand what the
>> difference to the default profile is and then you have to include your
>> dependencies again for the profile
>> - The pom comments are confusing

[Robert Metzger]

Hi,

I'm against cancelling a release for something that is not nice ;) It has
to be at least broken to cancel :)

I agree that the pom looks complicated and I would love to do it better,
but in my opinion the current approach is giving our users the best out of
the box experience.

The right approach of creating a Flink fat jar would be using the
maven-shade-plugin with the Flink dependencies set to "provided". This way
we tell the shade plugin that it can assume the core flink code to be
available. So there is no need to package those classes into the fat-jar.

The problem is that IntelliJ is not adding "provided" classes into the
classpath when importing the pom. So IntelliJ users will not be able to run
Flink jobs out of the IDE.

That's why the Flink dependencies are in the default scope.
The exclusions are in the maven-shade-plugin to reduce the jar size as much
as possible.

But there is also a maven profile to set the Flink dependencies to
provided, making the resulting jar as small as possible.

By the way, it is not possible to just exclude everything from
"org.apache.flink", because
a) users sometimes put their code into that package, so we would exclude
the code
b) Libraries like Gelly of Flink ML are also in the "org.apache.flink"
namespace but not provided on the server.

There is an ongoing discussion in IntelliJ's issue tracker whether to
change the behavior: https://youtrack.jetbrains.com/issue/IDEA-107048


Best,
Robert

On Thu, Jun 18, 2015 at 6:46 PM, Chiwan Park <[hidden email]> wrote:

> Is it okay when the user runs the built jar in LocalEnvironment? (Just run
> with `java -jar` command.)
> I know that it is special case but it is a possible scenario for local
> testing.
>
> If we change Quickstart POM to use provided scope for dependencies, we
> should add a guide about this into document.
>
> Regards,
> Chiwan Park
>
> > On Jun 19, 2015, at 12:53 AM, Aljoscha Krettek <[hidden email]>
> wrote:
> >
> > I'm also for simplification but let's hear what those who put the
> build-jar
> > profile there have to say about it.?
> >
> > On Thu, 18 Jun 2015 at 17:25 Ufuk Celebi <[hidden email]> wrote:
> >
> >>
> >> On 18 Jun 2015, at 16:58, Fabian Hueske <[hidden email]> wrote:
> >>
> >>> Why?
> >>>
> >>> mvn package
> >>>
> >>> builds the program correctly, no?
> >>
> >> Yes, but:
> >>
> >> - Dependencies not specified by the user may be included (Metrics,
> >> javaassist)
> >> - Dependencies specified by the user may be excluded
> >> - If you use the build-jar profile you have to understand what the
> >> difference to the default profile is and then you have to include your
> >> dependencies again for the profile
> >> - The pom comments are confusing
>
>
>
>
>

[Stephan Ewen]

I remember we had a similar discussion a while back.

The current solution is the best out-of-the-box experience for the
quickstarts across Eclipse, IntelliJ, and command-line Maven. This is more
important that nice and compact POM files, IMHO.



On Thu, Jun 18, 2015 at 11:26 AM, Robert Metzger <[hidden email]>
wrote:

> Hi,
>
> I'm against cancelling a release for something that is not nice ;) It has
> to be at least broken to cancel :)
>
> I agree that the pom looks complicated and I would love to do it better,
> but in my opinion the current approach is giving our users the best out of
> the box experience.
>
> The right approach of creating a Flink fat jar would be using the
> maven-shade-plugin with the Flink dependencies set to "provided". This way
> we tell the shade plugin that it can assume the core flink code to be
> available. So there is no need to package those classes into the fat-jar.
>
> The problem is that IntelliJ is not adding "provided" classes into the
> classpath when importing the pom. So IntelliJ users will not be able to run
> Flink jobs out of the IDE.
>
> That's why the Flink dependencies are in the default scope.
> The exclusions are in the maven-shade-plugin to reduce the jar size as much
> as possible.
>
> But there is also a maven profile to set the Flink dependencies to
> provided, making the resulting jar as small as possible.
>
> By the way, it is not possible to just exclude everything from
> "org.apache.flink", because
> a) users sometimes put their code into that package, so we would exclude
> the code
> b) Libraries like Gelly of Flink ML are also in the "org.apache.flink"
> namespace but not provided on the server.
>
> There is an ongoing discussion in IntelliJ's issue tracker whether to
> change the behavior: https://youtrack.jetbrains.com/issue/IDEA-107048
>
>
> Best,
> Robert
>
> On Thu, Jun 18, 2015 at 6:46 PM, Chiwan Park <[hidden email]>
> wrote:
>
> > Is it okay when the user runs the built jar in LocalEnvironment? (Just
> run
> > with `java -jar` command.)
> > I know that it is special case but it is a possible scenario for local
> > testing.
> >
> > If we change Quickstart POM to use provided scope for dependencies, we
> > should add a guide about this into document.
> >
> > Regards,
> > Chiwan Park
> >
> > > On Jun 19, 2015, at 12:53 AM, Aljoscha Krettek <[hidden email]>
> > wrote:
> > >
> > > I'm also for simplification but let's hear what those who put the
> > build-jar
> > > profile there have to say about it.?
> > >
> > > On Thu, 18 Jun 2015 at 17:25 Ufuk Celebi <[hidden email]> wrote:
> > >
> > >>
> > >> On 18 Jun 2015, at 16:58, Fabian Hueske <[hidden email]> wrote:
> > >>
> > >>> Why?
> > >>>
> > >>> mvn package
> > >>>
> > >>> builds the program correctly, no?
> > >>
> > >> Yes, but:
> > >>
> > >> - Dependencies not specified by the user may be included (Metrics,
> > >> javaassist)
> > >> - Dependencies specified by the user may be excluded
> > >> - If you use the build-jar profile you have to understand what the
> > >> difference to the default profile is and then you have to include your
> > >> dependencies again for the profile
> > >> - The pom comments are confusing
> >
> >
> >
> >
> >
>

[Ufuk Celebi-2]

On Thursday, June 18, 2015, Robert Metzger <[hidden email]> wrote:

> Hi,
>
> I'm against cancelling a release for something that is not nice ;) It has
> to be at least broken to cancel :)


See vote thread +1. This was not about canceling but clearification. ;)

> I agree that the pom looks complicated and I would love to do it better,
> but in my opinion the current approach is giving our users the best out of
> the box experience.
>
> The right approach of creating a Flink fat jar would be using the
> maven-shade-plugin with the Flink dependencies set to "provided". This way
> we tell the shade plugin that it can assume the core flink code to be
> available. So there is no need to package those classes into the fat-jar.
>
> The problem is that IntelliJ is not adding "provided" classes into the
> classpath when importing the pom. So IntelliJ users will not be able to run
> Flink jobs out of the IDE.

Ok this is too bad :(

[mxm]

@Robert @Stephan Thanks for clarifying! Of course it would be better to
have a concise quickstart pom.xml but the necessary workarounds justify the
current state.

On Thu, Jun 18, 2015 at 11:48 PM, Stephan Ewen <[hidden email]> wrote:

> I remember we had a similar discussion a while back.
>
> The current solution is the best out-of-the-box experience for the
> quickstarts across Eclipse, IntelliJ, and command-line Maven. This is more
> important that nice and compact POM files, IMHO.
>
>
>
> On Thu, Jun 18, 2015 at 11:26 AM, Robert Metzger <[hidden email]>
> wrote:
>
> > Hi,
> >
> > I'm against cancelling a release for something that is not nice ;) It has
> > to be at least broken to cancel :)
> >
> > I agree that the pom looks complicated and I would love to do it better,
> > but in my opinion the current approach is giving our users the best out
> of
> > the box experience.
> >
> > The right approach of creating a Flink fat jar would be using the
> > maven-shade-plugin with the Flink dependencies set to "provided". This
> way
> > we tell the shade plugin that it can assume the core flink code to be
> > available. So there is no need to package those classes into the fat-jar.
> >
> > The problem is that IntelliJ is not adding "provided" classes into the
> > classpath when importing the pom. So IntelliJ users will not be able to
> run
> > Flink jobs out of the IDE.
> >
> > That's why the Flink dependencies are in the default scope.
> > The exclusions are in the maven-shade-plugin to reduce the jar size as
> much
> > as possible.
> >
> > But there is also a maven profile to set the Flink dependencies to
> > provided, making the resulting jar as small as possible.
> >
> > By the way, it is not possible to just exclude everything from
> > "org.apache.flink", because
> > a) users sometimes put their code into that package, so we would exclude
> > the code
> > b) Libraries like Gelly of Flink ML are also in the "org.apache.flink"
> > namespace but not provided on the server.
> >
> > There is an ongoing discussion in IntelliJ's issue tracker whether to
> > change the behavior: https://youtrack.jetbrains.com/issue/IDEA-107048
> >
> >
> > Best,
> > Robert
> >
> > On Thu, Jun 18, 2015 at 6:46 PM, Chiwan Park <[hidden email]>
> > wrote:
> >
> > > Is it okay when the user runs the built jar in LocalEnvironment? (Just
> > run
> > > with `java -jar` command.)
> > > I know that it is special case but it is a possible scenario for local
> > > testing.
> > >
> > > If we change Quickstart POM to use provided scope for dependencies, we
> > > should add a guide about this into document.
> > >
> > > Regards,
> > > Chiwan Park
> > >
> > > > On Jun 19, 2015, at 12:53 AM, Aljoscha Krettek <[hidden email]>
> > > wrote:
> > > >
> > > > I'm also for simplification but let's hear what those who put the
> > > build-jar
> > > > profile there have to say about it.?
> > > >
> > > > On Thu, 18 Jun 2015 at 17:25 Ufuk Celebi <[hidden email]> wrote:
> > > >
> > > >>
> > > >> On 18 Jun 2015, at 16:58, Fabian Hueske <[hidden email]> wrote:
> > > >>
> > > >>> Why?
> > > >>>
> > > >>> mvn package
> > > >>>
> > > >>> builds the program correctly, no?
> > > >>
> > > >> Yes, but:
> > > >>
> > > >> - Dependencies not specified by the user may be included (Metrics,
> > > >> javaassist)
> > > >> - Dependencies specified by the user may be excluded
> > > >> - If you use the build-jar profile you have to understand what the
> > > >> difference to the default profile is and then you have to include
> your
> > > >> dependencies again for the profile
> > > >> - The pom comments are confusing
> > >
> > >
> > >
> > >
> > >
> >
>

[Ufuk Celebi-2]

On 19 Jun 2015, at 10:19, Maximilian Michels <[hidden email]> wrote:

> @Robert @Stephan Thanks for clarifying! Of course it would be better to
> have a concise quickstart pom.xml but the necessary workarounds justify the
> current state.

+1