flink-dist packaging including unshaded classes

[Nick Dimiduk-2]

Hi there,

I'm attempting to build locally a flink based on release-0.10.0 +
FLINK-3147. When I build from this sandbox, the resulting flink-dist.jar
contains both shanded and unshaded jars. In my case, this results in a
runtime conflict in my application, where com.google.common.base.Stopwatch
from both Guava-12 and Guava-18 are in my classpath.

Is there some additional profile required to build a dist package with only
the shaded jars?

Thanks,
Nick

$ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
$ cd flink-0.10.0
$ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
    testing:
org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
    testing: org/apache/flink/shaded/com/google/common/base/Stopwatch.class
  OK
    testing:
org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
  OK
    testing:
org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
  OK
    testing:
org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class   OK
    testing:
org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class   OK
    testing: com/google/inject/internal/util/$Stopwatch.class   OK

vs.

$ git status
HEAD detached from release-0.10.0
$ git log --decorate=short --oneline | head -n3
dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose fields as
protected
ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
c0fe305 [FLINK-2992] Remove use of SerializationUtils
$ mvn clean install -DskipTests
...
$ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
$ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
    testing:
org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
    testing: org/apache/flink/shaded/com/google/common/base/Stopwatch.class
  OK
    testing:
org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
  OK
    testing:
org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
  OK
    testing:
org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class   OK
    testing:
org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class   OK
    testing: com/google/inject/internal/util/$Stopwatch.class   OK
    testing: com/google/common/base/Stopwatch$1.class   OK
    testing: com/google/common/base/Stopwatch.class   OK
    testing:
com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class   OK
    testing:
com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class   OK

[Stephan Ewen]

Hi!

Did you change anything in the POM files, with respect to Guava, or add
another dependency that might transitively pull Guava?

Stephan


On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <[hidden email]> wrote:

> Hi there,
>
> I'm attempting to build locally a flink based on release-0.10.0 +
> FLINK-3147. When I build from this sandbox, the resulting flink-dist.jar
> contains both shanded and unshaded jars. In my case, this results in a
> runtime conflict in my application, where com.google.common.base.Stopwatch
> from both Guava-12 and Guava-18 are in my classpath.
>
> Is there some additional profile required to build a dist package with only
> the shaded jars?
>
> Thanks,
> Nick
>
> $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> $ cd flink-0.10.0
> $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
>     testing:
> org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
>     testing: org/apache/flink/shaded/com/google/common/base/Stopwatch.class
>   OK
>     testing:
>
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
>   OK
>     testing:
>
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
>   OK
>     testing:
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
>  OK
>     testing:
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class   OK
>     testing: com/google/inject/internal/util/$Stopwatch.class   OK
>
> vs.
>
> $ git status
> HEAD detached from release-0.10.0
> $ git log --decorate=short --oneline | head -n3
> dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose fields as
> protected
> ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> c0fe305 [FLINK-2992] Remove use of SerializationUtils
> $ mvn clean install -DskipTests
> ...
> $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
>     testing:
> org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
>     testing: org/apache/flink/shaded/com/google/common/base/Stopwatch.class
>   OK
>     testing:
>
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
>   OK
>     testing:
>
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
>   OK
>     testing:
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
>  OK
>     testing:
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class   OK
>     testing: com/google/inject/internal/util/$Stopwatch.class   OK
>     testing: com/google/common/base/Stopwatch$1.class   OK
>     testing: com/google/common/base/Stopwatch.class   OK
>     testing:
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
>  OK
>     testing:
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class   OK
>

[Nick Dimiduk]

I did not. All I did was apply the PR from FLINK-3147. I thought perhaps
there's some command line incantation I'm missing.

On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]> wrote:

> Hi!
>
> Did you change anything in the POM files, with respect to Guava, or add
> another dependency that might transitively pull Guava?
>
> Stephan
>
>
> On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <[hidden email]> wrote:
>
> > Hi there,
> >
> > I'm attempting to build locally a flink based on release-0.10.0 +
> > FLINK-3147. When I build from this sandbox, the resulting flink-dist.jar
> > contains both shanded and unshaded jars. In my case, this results in a
> > runtime conflict in my application, where
> com.google.common.base.Stopwatch
> > from both Guava-12 and Guava-18 are in my classpath.
> >
> > Is there some additional profile required to build a dist package with
> only
> > the shaded jars?
> >
> > Thanks,
> > Nick
> >
> > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > $ cd flink-0.10.0
> > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> >     testing:
> > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
> >     testing:
> org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> >   OK
> >     testing:
> >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> >   OK
> >     testing:
> >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> >   OK
> >     testing:
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> >  OK
> >     testing:
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
>  OK
> >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> >
> > vs.
> >
> > $ git status
> > HEAD detached from release-0.10.0
> > $ git log --decorate=short --oneline | head -n3
> > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose fields
> as
> > protected
> > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > $ mvn clean install -DskipTests
> > ...
> > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> >     testing:
> > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
> >     testing:
> org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> >   OK
> >     testing:
> >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> >   OK
> >     testing:
> >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> >   OK
> >     testing:
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> >  OK
> >     testing:
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
>  OK
> >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> >     testing: com/google/common/base/Stopwatch$1.class   OK
> >     testing: com/google/common/base/Stopwatch.class   OK
> >     testing:
> > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> >  OK
> >     testing:
> > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
>  OK
> >
>

[Stephan Ewen]

Usually, no command line magic is needed. Simple "mvn clean package" does
it.

May be that this is not related to your building at all. Can you check
whether the same is already the case on the master branch with Hadoop 2.7
Scala 2.11 ?

Also, simple way to diagnose where these dependencies come from is to do
inside the "flink-dist" project a "mvn dependency:tree" run. That shows how
the unshaded Guava was pulled in.

Greetings,
Stephan


On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]> wrote:

> I did not. All I did was apply the PR from FLINK-3147. I thought perhaps
> there's some command line incantation I'm missing.
>
> On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]> wrote:
>
> > Hi!
> >
> > Did you change anything in the POM files, with respect to Guava, or add
> > another dependency that might transitively pull Guava?
> >
> > Stephan
> >
> >
> > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <[hidden email]>
> wrote:
> >
> > > Hi there,
> > >
> > > I'm attempting to build locally a flink based on release-0.10.0 +
> > > FLINK-3147. When I build from this sandbox, the resulting
> flink-dist.jar
> > > contains both shanded and unshaded jars. In my case, this results in a
> > > runtime conflict in my application, where
> > com.google.common.base.Stopwatch
> > > from both Guava-12 and Guava-18 are in my classpath.
> > >
> > > Is there some additional profile required to build a dist package with
> > only
> > > the shaded jars?
> > >
> > > Thanks,
> > > Nick
> > >
> > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > $ cd flink-0.10.0
> > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> > >     testing:
> > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
> > >     testing:
> > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > >   OK
> > >     testing:
> > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > >   OK
> > >     testing:
> > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > >   OK
> > >     testing:
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > >  OK
> > >     testing:
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> >  OK
> > >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> > >
> > > vs.
> > >
> > > $ git status
> > > HEAD detached from release-0.10.0
> > > $ git log --decorate=short --oneline | head -n3
> > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose fields
> > as
> > > protected
> > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > $ mvn clean install -DskipTests
> > > ...
> > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > >     testing:
> > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
> > >     testing:
> > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > >   OK
> > >     testing:
> > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > >   OK
> > >     testing:
> > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > >   OK
> > >     testing:
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > >  OK
> > >     testing:
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> >  OK
> > >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > >     testing: com/google/common/base/Stopwatch.class   OK
> > >     testing:
> > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > >  OK
> > >     testing:
> > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> >  OK
> > >
> >
>

[Nick Dimiduk-2]

mvn dependency:tree from flink-dist module does not include any mention of
guava. When I build (mvn clean package -DskipTests) vs master (fc8be1c) I
see the same packaging problem.

On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]> wrote:

> Usually, no command line magic is needed. Simple "mvn clean package" does
> it.
>
> May be that this is not related to your building at all. Can you check
> whether the same is already the case on the master branch with Hadoop 2.7
> Scala 2.11 ?
>
> Also, simple way to diagnose where these dependencies come from is to do
> inside the "flink-dist" project a "mvn dependency:tree" run. That shows how
> the unshaded Guava was pulled in.
>
> Greetings,
> Stephan
>
>
> On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]> wrote:
>
> > I did not. All I did was apply the PR from FLINK-3147. I thought perhaps
> > there's some command line incantation I'm missing.
> >
> > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Hi!
> > >
> > > Did you change anything in the POM files, with respect to Guava, or add
> > > another dependency that might transitively pull Guava?
> > >
> > > Stephan
> > >
> > >
> > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <[hidden email]>
> > wrote:
> > >
> > > > Hi there,
> > > >
> > > > I'm attempting to build locally a flink based on release-0.10.0 +
> > > > FLINK-3147. When I build from this sandbox, the resulting
> > flink-dist.jar
> > > > contains both shanded and unshaded jars. In my case, this results in
> a
> > > > runtime conflict in my application, where
> > > com.google.common.base.Stopwatch
> > > > from both Guava-12 and Guava-18 are in my classpath.
> > > >
> > > > Is there some additional profile required to build a dist package
> with
> > > only
> > > > the shaded jars?
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > $ cd flink-0.10.0
> > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> > > >     testing:
> > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
> > > >     testing:
> > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > >   OK
> > > >     testing:
> > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > >   OK
> > > >     testing:
> > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > >   OK
> > > >     testing:
> > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > >  OK
> > > >     testing:
> > > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > >  OK
> > > >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> > > >
> > > > vs.
> > > >
> > > > $ git status
> > > > HEAD detached from release-0.10.0
> > > > $ git log --decorate=short --oneline | head -n3
> > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose
> fields
> > > as
> > > > protected
> > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > $ mvn clean install -DskipTests
> > > > ...
> > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > >     testing:
> > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class   OK
> > > >     testing:
> > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > >   OK
> > > >     testing:
> > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > >   OK
> > > >     testing:
> > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > >   OK
> > > >     testing:
> > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > >  OK
> > > >     testing:
> > > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > >  OK
> > > >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> > > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > >     testing:
> > > >
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > >  OK
> > > >     testing:
> > > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > >  OK
> > > >
> > >
> >
>

[Robert Metzger]

I can confirm that guava is part of the fat jar for the 2.7.0, scala 2.11
distribution.

I'll look into the issue tomorrow

On Wed, Dec 9, 2015 at 7:58 PM, Nick Dimiduk <[hidden email]> wrote:

> mvn dependency:tree from flink-dist module does not include any mention of
> guava. When I build (mvn clean package -DskipTests) vs master (fc8be1c) I
> see the same packaging problem.
>
> On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]> wrote:
>
> > Usually, no command line magic is needed. Simple "mvn clean package" does
> > it.
> >
> > May be that this is not related to your building at all. Can you check
> > whether the same is already the case on the master branch with Hadoop 2.7
> > Scala 2.11 ?
> >
> > Also, simple way to diagnose where these dependencies come from is to do
> > inside the "flink-dist" project a "mvn dependency:tree" run. That shows
> how
> > the unshaded Guava was pulled in.
> >
> > Greetings,
> > Stephan
> >
> >
> > On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]> wrote:
> >
> > > I did not. All I did was apply the PR from FLINK-3147. I thought
> perhaps
> > > there's some command line incantation I'm missing.
> > >
> > > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]> wrote:
> > >
> > > > Hi!
> > > >
> > > > Did you change anything in the POM files, with respect to Guava, or
> add
> > > > another dependency that might transitively pull Guava?
> > > >
> > > > Stephan
> > > >
> > > >
> > > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi there,
> > > > >
> > > > > I'm attempting to build locally a flink based on release-0.10.0 +
> > > > > FLINK-3147. When I build from this sandbox, the resulting
> > > flink-dist.jar
> > > > > contains both shanded and unshaded jars. In my case, this results
> in
> > a
> > > > > runtime conflict in my application, where
> > > > com.google.common.base.Stopwatch
> > > > > from both Guava-12 and Guava-18 are in my classpath.
> > > > >
> > > > > Is there some additional profile required to build a dist package
> > with
> > > > only
> > > > > the shaded jars?
> > > > >
> > > > > Thanks,
> > > > > Nick
> > > > >
> > > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > > $ cd flink-0.10.0
> > > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> > > > >     testing:
> > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
>  OK
> > > > >     testing:
> > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > >   OK
> > > > >     testing:
> > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > >   OK
> > > > >     testing:
> > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > >   OK
> > > > >     testing:
> > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > >  OK
> > > > >     testing:
> > > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > >  OK
> > > > >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> > > > >
> > > > > vs.
> > > > >
> > > > > $ git status
> > > > > HEAD detached from release-0.10.0
> > > > > $ git log --decorate=short --oneline | head -n3
> > > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose
> > fields
> > > > as
> > > > > protected
> > > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > > $ mvn clean install -DskipTests
> > > > > ...
> > > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > > >     testing:
> > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
>  OK
> > > > >     testing:
> > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > >   OK
> > > > >     testing:
> > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > >   OK
> > > > >     testing:
> > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > >   OK
> > > > >     testing:
> > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > >  OK
> > > > >     testing:
> > > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > >  OK
> > > > >     testing: com/google/inject/internal/util/$Stopwatch.class   OK
> > > > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > > >     testing:
> > > > >
> > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > >  OK
> > > > >     testing:
> > > > >
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > >  OK
> > > > >
> > > >
> > >
> >
>

[Nick Dimiduk-2]

Thanks, I appreciate it.

On Wed, Dec 9, 2015 at 12:50 PM, Robert Metzger <[hidden email]> wrote:

> I can confirm that guava is part of the fat jar for the 2.7.0, scala 2.11
> distribution.
>
> I'll look into the issue tomorrow
>
> On Wed, Dec 9, 2015 at 7:58 PM, Nick Dimiduk <[hidden email]> wrote:
>
> > mvn dependency:tree from flink-dist module does not include any mention
> of
> > guava. When I build (mvn clean package -DskipTests) vs master (fc8be1c) I
> > see the same packaging problem.
> >
> > On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Usually, no command line magic is needed. Simple "mvn clean package"
> does
> > > it.
> > >
> > > May be that this is not related to your building at all. Can you check
> > > whether the same is already the case on the master branch with Hadoop
> 2.7
> > > Scala 2.11 ?
> > >
> > > Also, simple way to diagnose where these dependencies come from is to
> do
> > > inside the "flink-dist" project a "mvn dependency:tree" run. That shows
> > how
> > > the unshaded Guava was pulled in.
> > >
> > > Greetings,
> > > Stephan
> > >
> > >
> > > On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]>
> wrote:
> > >
> > > > I did not. All I did was apply the PR from FLINK-3147. I thought
> > perhaps
> > > > there's some command line incantation I'm missing.
> > > >
> > > > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]>
> wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > Did you change anything in the POM files, with respect to Guava, or
> > add
> > > > > another dependency that might transitively pull Guava?
> > > > >
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Hi there,
> > > > > >
> > > > > > I'm attempting to build locally a flink based on release-0.10.0 +
> > > > > > FLINK-3147. When I build from this sandbox, the resulting
> > > > flink-dist.jar
> > > > > > contains both shanded and unshaded jars. In my case, this results
> > in
> > > a
> > > > > > runtime conflict in my application, where
> > > > > com.google.common.base.Stopwatch
> > > > > > from both Guava-12 and Guava-18 are in my classpath.
> > > > > >
> > > > > > Is there some additional profile required to build a dist package
> > > with
> > > > > only
> > > > > > the shaded jars?
> > > > > >
> > > > > > Thanks,
> > > > > > Nick
> > > > > >
> > > > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > > > $ cd flink-0.10.0
> > > > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> > > > > >     testing:
> > > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> >  OK
> > > > > >     testing:
> > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > >   OK
> > > > > >     testing:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > >   OK
> > > > > >     testing:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > >   OK
> > > > > >     testing:
> > > > > >
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > >  OK
> > > > > >     testing:
> > > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > >  OK
> > > > > >     testing: com/google/inject/internal/util/$Stopwatch.class
>  OK
> > > > > >
> > > > > > vs.
> > > > > >
> > > > > > $ git status
> > > > > > HEAD detached from release-0.10.0
> > > > > > $ git log --decorate=short --oneline | head -n3
> > > > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should expose
> > > fields
> > > > > as
> > > > > > protected
> > > > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > > > $ mvn clean install -DskipTests
> > > > > > ...
> > > > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > > > >     testing:
> > > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> >  OK
> > > > > >     testing:
> > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > >   OK
> > > > > >     testing:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > >   OK
> > > > > >     testing:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > >   OK
> > > > > >     testing:
> > > > > >
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > >  OK
> > > > > >     testing:
> > > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > >  OK
> > > > > >     testing: com/google/inject/internal/util/$Stopwatch.class
>  OK
> > > > > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > > > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > > > >     testing:
> > > > > >
> > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > >  OK
> > > > > >     testing:
> > > > > >
> > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > >  OK
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Robert Metzger]

I further looked into the issue. I have the strong feeling its a bug in
Maven .. or at least a change in its behavior.
The build with maven 3.2.5 is correct, with 3.3.9, the first build is
incorrect, the second build of "flink-dist" only.

This is the issue: https://issues.apache.org/jira/browse/FLINK-3158

For you as a workaround:

wget
http://archive.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
and then use that maven for now ;)


On Thu, Dec 10, 2015 at 12:35 AM, Nick Dimiduk <[hidden email]> wrote:

> Thanks, I appreciate it.
>
> On Wed, Dec 9, 2015 at 12:50 PM, Robert Metzger <[hidden email]>
> wrote:
>
> > I can confirm that guava is part of the fat jar for the 2.7.0, scala 2.11
> > distribution.
> >
> > I'll look into the issue tomorrow
> >
> > On Wed, Dec 9, 2015 at 7:58 PM, Nick Dimiduk <[hidden email]>
> wrote:
> >
> > > mvn dependency:tree from flink-dist module does not include any mention
> > of
> > > guava. When I build (mvn clean package -DskipTests) vs master
> (fc8be1c) I
> > > see the same packaging problem.
> > >
> > > On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]> wrote:
> > >
> > > > Usually, no command line magic is needed. Simple "mvn clean package"
> > does
> > > > it.
> > > >
> > > > May be that this is not related to your building at all. Can you
> check
> > > > whether the same is already the case on the master branch with Hadoop
> > 2.7
> > > > Scala 2.11 ?
> > > >
> > > > Also, simple way to diagnose where these dependencies come from is to
> > do
> > > > inside the "flink-dist" project a "mvn dependency:tree" run. That
> shows
> > > how
> > > > the unshaded Guava was pulled in.
> > > >
> > > > Greetings,
> > > > Stephan
> > > >
> > > >
> > > > On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]>
> > wrote:
> > > >
> > > > > I did not. All I did was apply the PR from FLINK-3147. I thought
> > > perhaps
> > > > > there's some command line incantation I'm missing.
> > > > >
> > > > > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]>
> > wrote:
> > > > >
> > > > > > Hi!
> > > > > >
> > > > > > Did you change anything in the POM files, with respect to Guava,
> or
> > > add
> > > > > > another dependency that might transitively pull Guava?
> > > > > >
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Hi there,
> > > > > > >
> > > > > > > I'm attempting to build locally a flink based on
> release-0.10.0 +
> > > > > > > FLINK-3147. When I build from this sandbox, the resulting
> > > > > flink-dist.jar
> > > > > > > contains both shanded and unshaded jars. In my case, this
> results
> > > in
> > > > a
> > > > > > > runtime conflict in my application, where
> > > > > > com.google.common.base.Stopwatch
> > > > > > > from both Guava-12 and Guava-18 are in my classpath.
> > > > > > >
> > > > > > > Is there some additional profile required to build a dist
> package
> > > > with
> > > > > > only
> > > > > > > the shaded jars?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Nick
> > > > > > >
> > > > > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > > > > $ cd flink-0.10.0
> > > > > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> > > > > > >     testing:
> > > > > > >
> org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > >  OK
> > > > > > >     testing:
> > > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > >   OK
> > > > > > >     testing:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > >   OK
> > > > > > >     testing:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > >   OK
> > > > > > >     testing:
> > > > > > >
> > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > >  OK
> > > > > > >     testing:
> > > > > > >
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > >  OK
> > > > > > >     testing: com/google/inject/internal/util/$Stopwatch.class
> >  OK
> > > > > > >
> > > > > > > vs.
> > > > > > >
> > > > > > > $ git status
> > > > > > > HEAD detached from release-0.10.0
> > > > > > > $ git log --decorate=short --oneline | head -n3
> > > > > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should
> expose
> > > > fields
> > > > > > as
> > > > > > > protected
> > > > > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > > > > $ mvn clean install -DskipTests
> > > > > > > ...
> > > > > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > > > > >     testing:
> > > > > > >
> org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > >  OK
> > > > > > >     testing:
> > > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > >   OK
> > > > > > >     testing:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > >   OK
> > > > > > >     testing:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > >   OK
> > > > > > >     testing:
> > > > > > >
> > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > >  OK
> > > > > > >     testing:
> > > > > > >
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > >  OK
> > > > > > >     testing: com/google/inject/internal/util/$Stopwatch.class
> >  OK
> > > > > > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > > > > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > > > > >     testing:
> > > > > > >
> > > >
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > >  OK
> > > > > > >     testing:
> > > > > > >
> > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > >  OK
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Nick Dimiduk-2]

Lol. Okay, thanks a bunch. Mind linking back here with your discussion
thread on the maven list? This will be of interest to other projects as
well.

On Thursday, December 10, 2015, Robert Metzger <[hidden email]> wrote:

> I further looked into the issue. I have the strong feeling its a bug in
> Maven .. or at least a change in its behavior.
> The build with maven 3.2.5 is correct, with 3.3.9, the first build is
> incorrect, the second build of "flink-dist" only.
>
> This is the issue: https://issues.apache.org/jira/browse/FLINK-3158
>
> For you as a workaround:
>
> wget
>
> http://archive.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
> and then use that maven for now ;)
>
>
> On Thu, Dec 10, 2015 at 12:35 AM, Nick Dimiduk <[hidden email]
> <javascript:;>> wrote:
>
> > Thanks, I appreciate it.
> >
> > On Wed, Dec 9, 2015 at 12:50 PM, Robert Metzger <[hidden email]
> <javascript:;>>
> > wrote:
> >
> > > I can confirm that guava is part of the fat jar for the 2.7.0, scala
> 2.11
> > > distribution.
> > >
> > > I'll look into the issue tomorrow
> > >
> > > On Wed, Dec 9, 2015 at 7:58 PM, Nick Dimiduk <[hidden email]
> <javascript:;>>
> > wrote:
> > >
> > > > mvn dependency:tree from flink-dist module does not include any
> mention
> > > of
> > > > guava. When I build (mvn clean package -DskipTests) vs master
> > (fc8be1c) I
> > > > see the same packaging problem.
> > > >
> > > > On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]
> <javascript:;>> wrote:
> > > >
> > > > > Usually, no command line magic is needed. Simple "mvn clean
> package"
> > > does
> > > > > it.
> > > > >
> > > > > May be that this is not related to your building at all. Can you
> > check
> > > > > whether the same is already the case on the master branch with
> Hadoop
> > > 2.7
> > > > > Scala 2.11 ?
> > > > >
> > > > > Also, simple way to diagnose where these dependencies come from is
> to
> > > do
> > > > > inside the "flink-dist" project a "mvn dependency:tree" run. That
> > shows
> > > > how
> > > > > the unshaded Guava was pulled in.
> > > > >
> > > > > Greetings,
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]
> <javascript:;>>
> > > wrote:
> > > > >
> > > > > > I did not. All I did was apply the PR from FLINK-3147. I thought
> > > > perhaps
> > > > > > there's some command line incantation I'm missing.
> > > > > >
> > > > > > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]
> <javascript:;>>
> > > wrote:
> > > > > >
> > > > > > > Hi!
> > > > > > >
> > > > > > > Did you change anything in the POM files, with respect to
> Guava,
> > or
> > > > add
> > > > > > > another dependency that might transitively pull Guava?
> > > > > > >
> > > > > > > Stephan
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <
> > [hidden email] <javascript:;>>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi there,
> > > > > > > >
> > > > > > > > I'm attempting to build locally a flink based on
> > release-0.10.0 +
> > > > > > > > FLINK-3147. When I build from this sandbox, the resulting
> > > > > > flink-dist.jar
> > > > > > > > contains both shanded and unshaded jars. In my case, this
> > results
> > > > in
> > > > > a
> > > > > > > > runtime conflict in my application, where
> > > > > > > com.google.common.base.Stopwatch
> > > > > > > > from both Guava-12 and Guava-18 are in my classpath.
> > > > > > > >
> > > > > > > > Is there some additional profile required to build a dist
> > package
> > > > > with
> > > > > > > only
> > > > > > > > the shaded jars?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Nick
> > > > > > > >
> > > > > > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > > > > > $ cd flink-0.10.0
> > > > > > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep Stopwatch
> > > > > > > >     testing:
> > > > > > > >
> > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > > >  OK
> > > > > > > >     testing:
> > > > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > > >   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > >   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > >   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > > >  OK
> > > > > > > >     testing:
> > > > > > > >
> > > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > > >  OK
> > > > > > > >     testing: com/google/inject/internal/util/$Stopwatch.class
> > >  OK
> > > > > > > >
> > > > > > > > vs.
> > > > > > > >
> > > > > > > > $ git status
> > > > > > > > HEAD detached from release-0.10.0
> > > > > > > > $ git log --decorate=short --oneline | head -n3
> > > > > > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should
> > expose
> > > > > fields
> > > > > > > as
> > > > > > > > protected
> > > > > > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > > > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > > > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > > > > > $ mvn clean install -DskipTests
> > > > > > > > ...
> > > > > > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > > > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > > > > > >     testing:
> > > > > > > >
> > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > > >  OK
> > > > > > > >     testing:
> > > > > > > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > > >   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > >   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > >   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > > >  OK
> > > > > > > >     testing:
> > > > > > > >
> > > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > > >  OK
> > > > > > > >     testing: com/google/inject/internal/util/$Stopwatch.class
> > >  OK
> > > > > > > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > > > > > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > > > > > >     testing:
> > > > > > > >
> > > > >
> > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > >  OK
> > > > > > > >     testing:
> > > > > > > >
> > > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > >  OK
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Robert Metzger]

You are right. I'll post the link with my next message on the maven user
list.
Here is the link to the maven discussion:
http://mail-archives.apache.org/mod_mbox/maven-users/201512.mbox/%3CCAGr9p8CdMJL0sdbewgZ5WqJ0_4OWCox-mw00T7Vn3KDxz9PjtA%40mail.gmail.com%3E

His last answer says that Maven 3.2's behavior is considered buggy. I'm
wondering whether we should use the maven-enforcer-plugin to require Maven
3.3.
I guess many Linux distributions are still at Maven 3.2, so users might get
unhappy users


On Thu, Dec 10, 2015 at 6:33 PM, Nick Dimiduk <[hidden email]> wrote:

> Lol. Okay, thanks a bunch. Mind linking back here with your discussion
> thread on the maven list? This will be of interest to other projects as
> well.
>
> On Thursday, December 10, 2015, Robert Metzger <[hidden email]>
> wrote:
>
> > I further looked into the issue. I have the strong feeling its a bug in
> > Maven .. or at least a change in its behavior.
> > The build with maven 3.2.5 is correct, with 3.3.9, the first build is
> > incorrect, the second build of "flink-dist" only.
> >
> > This is the issue: https://issues.apache.org/jira/browse/FLINK-3158
> >
> > For you as a workaround:
> >
> > wget
> >
> >
> http://archive.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
> > and then use that maven for now ;)
> >
> >
> > On Thu, Dec 10, 2015 at 12:35 AM, Nick Dimiduk <[hidden email]
> > <javascript:;>> wrote:
> >
> > > Thanks, I appreciate it.
> > >
> > > On Wed, Dec 9, 2015 at 12:50 PM, Robert Metzger <[hidden email]
> > <javascript:;>>
> > > wrote:
> > >
> > > > I can confirm that guava is part of the fat jar for the 2.7.0, scala
> > 2.11
> > > > distribution.
> > > >
> > > > I'll look into the issue tomorrow
> > > >
> > > > On Wed, Dec 9, 2015 at 7:58 PM, Nick Dimiduk <[hidden email]
> > <javascript:;>>
> > > wrote:
> > > >
> > > > > mvn dependency:tree from flink-dist module does not include any
> > mention
> > > > of
> > > > > guava. When I build (mvn clean package -DskipTests) vs master
> > > (fc8be1c) I
> > > > > see the same packaging problem.
> > > > >
> > > > > On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]
> > <javascript:;>> wrote:
> > > > >
> > > > > > Usually, no command line magic is needed. Simple "mvn clean
> > package"
> > > > does
> > > > > > it.
> > > > > >
> > > > > > May be that this is not related to your building at all. Can you
> > > check
> > > > > > whether the same is already the case on the master branch with
> > Hadoop
> > > > 2.7
> > > > > > Scala 2.11 ?
> > > > > >
> > > > > > Also, simple way to diagnose where these dependencies come from
> is
> > to
> > > > do
> > > > > > inside the "flink-dist" project a "mvn dependency:tree" run. That
> > > shows
> > > > > how
> > > > > > the unshaded Guava was pulled in.
> > > > > >
> > > > > > Greetings,
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <[hidden email]
> > <javascript:;>>
> > > > wrote:
> > > > > >
> > > > > > > I did not. All I did was apply the PR from FLINK-3147. I
> thought
> > > > > perhaps
> > > > > > > there's some command line incantation I'm missing.
> > > > > > >
> > > > > > > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <[hidden email]
> > <javascript:;>>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > Did you change anything in the POM files, with respect to
> > Guava,
> > > or
> > > > > add
> > > > > > > > another dependency that might transitively pull Guava?
> > > > > > > >
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <
> > > [hidden email] <javascript:;>>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi there,
> > > > > > > > >
> > > > > > > > > I'm attempting to build locally a flink based on
> > > release-0.10.0 +
> > > > > > > > > FLINK-3147. When I build from this sandbox, the resulting
> > > > > > > flink-dist.jar
> > > > > > > > > contains both shanded and unshaded jars. In my case, this
> > > results
> > > > > in
> > > > > > a
> > > > > > > > > runtime conflict in my application, where
> > > > > > > > com.google.common.base.Stopwatch
> > > > > > > > > from both Guava-12 and Guava-18 are in my classpath.
> > > > > > > > >
> > > > > > > > > Is there some additional profile required to build a dist
> > > package
> > > > > > with
> > > > > > > > only
> > > > > > > > > the shaded jars?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Nick
> > > > > > > > >
> > > > > > > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > > > > > > $ cd flink-0.10.0
> > > > > > > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep
> Stopwatch
> > > > > > > > >     testing:
> > > > > > > > >
> > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > > > >  OK
> > > > > > > > >     testing:
> > > > > > > >
> org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > > > >   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > > >   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > > >   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > >
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > > > >  OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > > > >  OK
> > > > > > > > >     testing:
> com/google/inject/internal/util/$Stopwatch.class
> > > >  OK
> > > > > > > > >
> > > > > > > > > vs.
> > > > > > > > >
> > > > > > > > > $ git status
> > > > > > > > > HEAD detached from release-0.10.0
> > > > > > > > > $ git log --decorate=short --oneline | head -n3
> > > > > > > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should
> > > expose
> > > > > > fields
> > > > > > > > as
> > > > > > > > > protected
> > > > > > > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > > > > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > > > > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > > > > > > $ mvn clean install -DskipTests
> > > > > > > > > ...
> > > > > > > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > > > > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > > > > > > >     testing:
> > > > > > > > >
> > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > > > >  OK
> > > > > > > > >     testing:
> > > > > > > >
> org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > > > >   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > > >   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > > >   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > >
> > > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > > > >  OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > > > >  OK
> > > > > > > > >     testing:
> com/google/inject/internal/util/$Stopwatch.class
> > > >  OK
> > > > > > > > >     testing: com/google/common/base/Stopwatch$1.class   OK
> > > > > > > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > > >
> > > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > > >  OK
> > > > > > > > >     testing:
> > > > > > > > >
> > > > >
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > >  OK
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Nick Dimiduk-2]

Just to confirm, building with maven 3.2.5 appears to work as expected.
Thanks a lot for the work-around.

On Thu, Dec 10, 2015 at 10:23 AM, Robert Metzger <[hidden email]>
wrote:

> You are right. I'll post the link with my next message on the maven user
> list.
> Here is the link to the maven discussion:
>
> http://mail-archives.apache.org/mod_mbox/maven-users/201512.mbox/%3CCAGr9p8CdMJL0sdbewgZ5WqJ0_4OWCox-mw00T7Vn3KDxz9PjtA%40mail.gmail.com%3E
>
> His last answer says that Maven 3.2's behavior is considered buggy. I'm
> wondering whether we should use the maven-enforcer-plugin to require Maven
> 3.3.
> I guess many Linux distributions are still at Maven 3.2, so users might get
> unhappy users
>
>
> On Thu, Dec 10, 2015 at 6:33 PM, Nick Dimiduk <[hidden email]> wrote:
>
> > Lol. Okay, thanks a bunch. Mind linking back here with your discussion
> > thread on the maven list? This will be of interest to other projects as
> > well.
> >
> > On Thursday, December 10, 2015, Robert Metzger <[hidden email]>
> > wrote:
> >
> > > I further looked into the issue. I have the strong feeling its a bug in
> > > Maven .. or at least a change in its behavior.
> > > The build with maven 3.2.5 is correct, with 3.3.9, the first build is
> > > incorrect, the second build of "flink-dist" only.
> > >
> > > This is the issue: https://issues.apache.org/jira/browse/FLINK-3158
> > >
> > > For you as a workaround:
> > >
> > > wget
> > >
> > >
> >
> http://archive.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
> > > and then use that maven for now ;)
> > >
> > >
> > > On Thu, Dec 10, 2015 at 12:35 AM, Nick Dimiduk <[hidden email]
> > > <javascript:;>> wrote:
> > >
> > > > Thanks, I appreciate it.
> > > >
> > > > On Wed, Dec 9, 2015 at 12:50 PM, Robert Metzger <[hidden email]
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > I can confirm that guava is part of the fat jar for the 2.7.0,
> scala
> > > 2.11
> > > > > distribution.
> > > > >
> > > > > I'll look into the issue tomorrow
> > > > >
> > > > > On Wed, Dec 9, 2015 at 7:58 PM, Nick Dimiduk <[hidden email]
> > > <javascript:;>>
> > > > wrote:
> > > > >
> > > > > > mvn dependency:tree from flink-dist module does not include any
> > > mention
> > > > > of
> > > > > > guava. When I build (mvn clean package -DskipTests) vs master
> > > > (fc8be1c) I
> > > > > > see the same packaging problem.
> > > > > >
> > > > > > On Wed, Dec 9, 2015 at 9:29 AM, Stephan Ewen <[hidden email]
> > > <javascript:;>> wrote:
> > > > > >
> > > > > > > Usually, no command line magic is needed. Simple "mvn clean
> > > package"
> > > > > does
> > > > > > > it.
> > > > > > >
> > > > > > > May be that this is not related to your building at all. Can
> you
> > > > check
> > > > > > > whether the same is already the case on the master branch with
> > > Hadoop
> > > > > 2.7
> > > > > > > Scala 2.11 ?
> > > > > > >
> > > > > > > Also, simple way to diagnose where these dependencies come from
> > is
> > > to
> > > > > do
> > > > > > > inside the "flink-dist" project a "mvn dependency:tree" run.
> That
> > > > shows
> > > > > > how
> > > > > > > the unshaded Guava was pulled in.
> > > > > > >
> > > > > > > Greetings,
> > > > > > > Stephan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 9, 2015 at 6:22 PM, Nick Dimiduk <
> [hidden email]
> > > <javascript:;>>
> > > > > wrote:
> > > > > > >
> > > > > > > > I did not. All I did was apply the PR from FLINK-3147. I
> > thought
> > > > > > perhaps
> > > > > > > > there's some command line incantation I'm missing.
> > > > > > > >
> > > > > > > > On Wed, Dec 9, 2015 at 3:29 AM, Stephan Ewen <
> [hidden email]
> > > <javascript:;>>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi!
> > > > > > > > >
> > > > > > > > > Did you change anything in the POM files, with respect to
> > > Guava,
> > > > or
> > > > > > add
> > > > > > > > > another dependency that might transitively pull Guava?
> > > > > > > > >
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Dec 8, 2015 at 9:25 PM, Nick Dimiduk <
> > > > [hidden email] <javascript:;>>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi there,
> > > > > > > > > >
> > > > > > > > > > I'm attempting to build locally a flink based on
> > > > release-0.10.0 +
> > > > > > > > > > FLINK-3147. When I build from this sandbox, the resulting
> > > > > > > > flink-dist.jar
> > > > > > > > > > contains both shanded and unshaded jars. In my case, this
> > > > results
> > > > > > in
> > > > > > > a
> > > > > > > > > > runtime conflict in my application, where
> > > > > > > > > com.google.common.base.Stopwatch
> > > > > > > > > > from both Guava-12 and Guava-18 are in my classpath.
> > > > > > > > > >
> > > > > > > > > > Is there some additional profile required to build a dist
> > > > package
> > > > > > > with
> > > > > > > > > only
> > > > > > > > > > the shaded jars?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Nick
> > > > > > > > > >
> > > > > > > > > > $ tar xvzf flink-0.10.0-bin-hadoop27-scala_2.11.tgz
> > > > > > > > > > $ cd flink-0.10.0
> > > > > > > > > > $ $ unzip -t lib/flink-dist_2.11-0.10.0.jar | grep
> > Stopwatch
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > > > > >  OK
> > > > > > > > > >     testing:
> > > > > > > > >
> > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > > > > >   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > > > >   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > > > >   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > >
> > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > > > > >  OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > > > > >  OK
> > > > > > > > > >     testing:
> > com/google/inject/internal/util/$Stopwatch.class
> > > > >  OK
> > > > > > > > > >
> > > > > > > > > > vs.
> > > > > > > > > >
> > > > > > > > > > $ git status
> > > > > > > > > > HEAD detached from release-0.10.0
> > > > > > > > > > $ git log --decorate=short --oneline | head -n3
> > > > > > > > > > dccdbd8 (HEAD) [FLINK-3147] HadoopOutputFormatBase should
> > > > expose
> > > > > > > fields
> > > > > > > > > as
> > > > > > > > > > protected
> > > > > > > > > > ab2cca4 (tag: release-0.10.0, origin/release-0.10.0-rc8,
> > > > > > > > > > ndimiduk/release-0.10.0-rc8) Commit for release 0.10.0
> > > > > > > > > > c0fe305 [FLINK-2992] Remove use of SerializationUtils
> > > > > > > > > > $ mvn clean install -DskipTests
> > > > > > > > > > ...
> > > > > > > > > > $ cd flink-dist/target/flink-0.10.0-bin/flink-0.10.0
> > > > > > > > > > $ unzip -t lib/flink-dist-0.10.0.jar | grep Stopwatch
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > org/apache/flink/shaded/com/google/common/base/Stopwatch$1.class
> > > > > >  OK
> > > > > > > > > >     testing:
> > > > > > > > >
> > org/apache/flink/shaded/com/google/common/base/Stopwatch.class
> > > > > > > > > >   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > > > >   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org/apache/flink/shaded/com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > > > >   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > >
> > > >
> org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch$1.class
> > > > > > > > > >  OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > >
> > org/apache/flink/hadoop/shaded/com/google/common/base/Stopwatch.class
> > > > > > > > >  OK
> > > > > > > > > >     testing:
> > com/google/inject/internal/util/$Stopwatch.class
> > > > >  OK
> > > > > > > > > >     testing: com/google/common/base/Stopwatch$1.class
>  OK
> > > > > > > > > >     testing: com/google/common/base/Stopwatch.class   OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > > >
> > > >
> com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
> > > > > > > > > >  OK
> > > > > > > > > >     testing:
> > > > > > > > > >
> > > > > >
> > com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
> > > > > > > > >  OK
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>