Shading Guava Dependency

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Shading Guava Dependency

Stephan Ewen
Hi!

I have looked into Robert's pull request about the Guava shading.

The problem is that shading builds uberjars and adds all guava files to all
the jars created by the projects. That blows up the code base by a lot and
seems dirty:

We have the guava classes more than 10 times in the classpath in the end
and blow the lib flolder up by 50 MB.

I modified it a bit and now have the following:

1) There is a project "flink-shaded", which builds the jars with the
relocated guava dependency (now in "org.apache.flink.shaded.com.google").
We can later relocate also other libraries to that project.

2) The core and other projects depend in "flink-shaded" for their guava

3) All projects are processed by the shade plugin, which only re-writes the
guava namespace, but does not add any classes to the jars.

4) All projects that depend on guava need to be shaded as well, such that
we can exclude the guava dependency there...


What do you think?


Stephan
Reply | Threaded
Open this post in threaded view
|

Re: Shading Guava Dependency

Stephan Ewen
Update: This is the current solution:


1) A global "provided" dependency on guava --> guava is available to our
projects, but is not included in libs and is not re-exported as a dependency

2) A dedicated shading project that builds a jar with shaded libs
(overrides provided scope with "compile", but produces "dependency reduced
pom")

3) A global shade phase that rewrites our references to guava to the shaded
namespace

4) Core has a dependency to the shading project (to make sure maven
dependency resolution has always the shaded file in the classpath)

5) Streaming connectors (where transitive dependencies use guava) are
shaded as well


==> We have a single shaded guava version in the lib folder and in the
maven dependencies

==> All our code refers to that relocated version

==> Users do not see us providing Guava, they need to provide their own
guava dependency as part of the job jars


*Open questions:*

 - We redistribute guava in the shaded version through maven central,
because we are producing the "flink-shaded" artifact. With guava being ACL
2, it should be alright. But I'd like to have a second opinion on that.


Stephan





On Mon, Sep 29, 2014 at 6:04 PM, Stephan Ewen <[hidden email]> wrote:

> Hi!
>
> I have looked into Robert's pull request about the Guava shading.
>
> The problem is that shading builds uberjars and adds all guava files to
> all the jars created by the projects. That blows up the code base by a lot
> and seems dirty:
>
> We have the guava classes more than 10 times in the classpath in the end
> and blow the lib flolder up by 50 MB.
>
> I modified it a bit and now have the following:
>
> 1) There is a project "flink-shaded", which builds the jars with the
> relocated guava dependency (now in "org.apache.flink.shaded.com.google").
> We can later relocate also other libraries to that project.
>
> 2) The core and other projects depend in "flink-shaded" for their guava
>
> 3) All projects are processed by the shade plugin, which only re-writes
> the guava namespace, but does not add any classes to the jars.
>
> 4) All projects that depend on guava need to be shaded as well, such that
> we can exclude the guava dependency there...
>
>
> What do you think?
>
>
> Stephan
>
>