Flink Maven Configuration

[Sebastian Kunert]

Hi Guys,

I am currently working on integrating Flink with Mesos. Therefore I
currently reuse the flink-yarn uberjar assembly to build a jar that
contains all the classes I need. Now I have a dependency on a Google
Protobuf class. I added protobuf as a dependency to my module and it is
also contained in the uberjar under /com/google/protobuf.

Nevertheless, on execution I get the following error:
java.lang.ClassNotFoundException:
org.apache.flink.shaded.com.google.protobuf.ByteString

The location org.apache.flink.shaded... is wrong. I understand this is
related to the maven shading plugin but after some looking into it I am not
sure how to fix this.

Greetings,

Sebastian

[Stephan Ewen]

Hey Sebastian!

I think there are two approaches:

  - You can either shade the protobuf Classes as well

  - or you can exclude the ptotobuf namespace from shading. Have a look at
the shade plugin docs.

Stephan
 Am 14.11.2014 09:49 schrieb "Sebastian Kunert" <[hidden email]>:

> Hi Guys,
>
> I am currently working on integrating Flink with Mesos. Therefore I
> currently reuse the flink-yarn uberjar assembly to build a jar that
> contains all the classes I need. Now I have a dependency on a Google
> Protobuf class. I added protobuf as a dependency to my module and it is
> also contained in the uberjar under /com/google/protobuf.
>
> Nevertheless, on execution I get the following error:
> java.lang.ClassNotFoundException:
> org.apache.flink.shaded.com.google.protobuf.ByteString
>
> The location org.apache.flink.shaded... is wrong. I understand this is
> related to the maven shading plugin but after some looking into it I am not
> sure how to fix this.
>
> Greetings,
>
> Sebastian
>

[Sebastian Kunert]

Thanks for your answer.

I did already take a look at the documentation. We have the following code
snippet in our pom in flink-shade.

<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>org.apache.flink.shaded.com.google</shadedPattern>
</relocation>
</relocations>

I first thought that the includes specify a whitelist of the packages to be
relocated, so from my understanding it would not be necessary to exclude
protobuf explicitly. Nevertheless, even the following change doesn't work.

<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
<excludes>
<exclude>com.google.protobuf:*</exclude>
</excludes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>org.apache.flink.shaded.com.google</shadedPattern>
</relocation>
</relocations>

 Making the relocation part more specific solves the problem.

<relocation>
<pattern>com.google.guava</pattern>
<shadedPattern>org.apache.flink.shaded.com.google.guava</shadedPattern>
</relocation>


2014-11-14 10:22 GMT+01:00 Stephan Ewen <[hidden email]>:

> Hey Sebastian!
>
> I think there are two approaches:
>
>   - You can either shade the protobuf Classes as well
>
>   - or you can exclude the ptotobuf namespace from shading. Have a look at
> the shade plugin docs.
>
> Stephan
>  Am 14.11.2014 09:49 schrieb "Sebastian Kunert" <[hidden email]>:
>
> > Hi Guys,
> >
> > I am currently working on integrating Flink with Mesos. Therefore I
> > currently reuse the flink-yarn uberjar assembly to build a jar that
> > contains all the classes I need. Now I have a dependency on a Google
> > Protobuf class. I added protobuf as a dependency to my module and it is
> > also contained in the uberjar under /com/google/protobuf.
> >
> > Nevertheless, on execution I get the following error:
> > java.lang.ClassNotFoundException:
> > org.apache.flink.shaded.com.google.protobuf.ByteString
> >
> > The location org.apache.flink.shaded... is wrong. I understand this is
> > related to the maven shading plugin but after some looking into it I am
> not
> > sure how to fix this.
> >
> > Greetings,
> >
> > Sebastian
> >
>

[Stephan Ewen]

I think that basically destroys the guava shading, because the guava
classes are not in "com.google.guava", rather than in "com.google.common"
and "com.google.thirdparty"...

The excludes you defined refer to what code goes into the shaded jar, not
what is relocated. If you define the excludes in the relocation, it should
work.

Stephan


On Fri, Nov 14, 2014 at 1:36 PM, Sebastian Kunert <[hidden email]>
wrote:

> Thanks for your answer.
>
> I did already take a look at the documentation. We have the following code
> snippet in our pom in flink-shade.
>
> <artifactSet>
> <includes>
> <include>com.google.guava:guava</include>
> </includes>
> </artifactSet>
> <relocations>
> <relocation>
> <pattern>com.google</pattern>
> <shadedPattern>org.apache.flink.shaded.com.google</shadedPattern>
> </relocation>
> </relocations>
>
> I first thought that the includes specify a whitelist of the packages to be
> relocated, so from my understanding it would not be necessary to exclude
> protobuf explicitly. Nevertheless, even the following change doesn't work.
>
> <artifactSet>
> <includes>
> <include>com.google.guava:guava</include>
> </includes>
> <excludes>
> <exclude>com.google.protobuf:*</exclude>
> </excludes>
> </artifactSet>
> <relocations>
> <relocation>
> <pattern>com.google</pattern>
> <shadedPattern>org.apache.flink.shaded.com.google</shadedPattern>
> </relocation>
> </relocations>
>
>  Making the relocation part more specific solves the problem.
>
> <relocation>
> <pattern>com.google.guava</pattern>
> <shadedPattern>org.apache.flink.shaded.com.google.guava</shadedPattern>
> </relocation>
>
>
> 2014-11-14 10:22 GMT+01:00 Stephan Ewen <[hidden email]>:
>
> > Hey Sebastian!
> >
> > I think there are two approaches:
> >
> >   - You can either shade the protobuf Classes as well
> >
> >   - or you can exclude the ptotobuf namespace from shading. Have a look
> at
> > the shade plugin docs.
> >
> > Stephan
> >  Am 14.11.2014 09:49 schrieb "Sebastian Kunert" <[hidden email]>:
> >
> > > Hi Guys,
> > >
> > > I am currently working on integrating Flink with Mesos. Therefore I
> > > currently reuse the flink-yarn uberjar assembly to build a jar that
> > > contains all the classes I need. Now I have a dependency on a Google
> > > Protobuf class. I added protobuf as a dependency to my module and it is
> > > also contained in the uberjar under /com/google/protobuf.
> > >
> > > Nevertheless, on execution I get the following error:
> > > java.lang.ClassNotFoundException:
> > > org.apache.flink.shaded.com.google.protobuf.ByteString
> > >
> > > The location org.apache.flink.shaded... is wrong. I understand this is
> > > related to the maven shading plugin but after some looking into it I am
> > not
> > > sure how to fix this.
> > >
> > > Greetings,
> > >
> > > Sebastian
> > >
> >
>

[Sebastian Kunert]

Ah of course. That did the trick, thank you.

Stephan Ewen <[hidden email]> schrieb am Fri Nov 14 2014 at 14:46:51:

> I think that basically destroys the guava shading, because the guava
> classes are not in "com.google.guava", rather than in "com.google.common"
> and "com.google.thirdparty"...
>
> The excludes you defined refer to what code goes into the shaded jar, not
> what is relocated. If you define the excludes in the relocation, it should
> work.
>
> Stephan
>
>
> On Fri, Nov 14, 2014 at 1:36 PM, Sebastian Kunert <[hidden email]>
> wrote:
>
> > Thanks for your answer.
> >
> > I did already take a look at the documentation. We have the following
> code
> > snippet in our pom in flink-shade.
> >
> > <artifactSet>
> > <includes>
> > <include>com.google.guava:guava</include>
> > </includes>
> > </artifactSet>
> > <relocations>
> > <relocation>
> > <pattern>com.google</pattern>
> > <shadedPattern>org.apache.flink.shaded.com.google</shadedPattern>
> > </relocation>
> > </relocations>
> >
> > I first thought that the includes specify a whitelist of the packages to
> be
> > relocated, so from my understanding it would not be necessary to exclude
> > protobuf explicitly. Nevertheless, even the following change doesn't
> work.
> >
> > <artifactSet>
> > <includes>
> > <include>com.google.guava:guava</include>
> > </includes>
> > <excludes>
> > <exclude>com.google.protobuf:*</exclude>
> > </excludes>
> > </artifactSet>
> > <relocations>
> > <relocation>
> > <pattern>com.google</pattern>
> > <shadedPattern>org.apache.flink.shaded.com.google</shadedPattern>
> > </relocation>
> > </relocations>
> >
> >  Making the relocation part more specific solves the problem.
> >
> > <relocation>
> > <pattern>com.google.guava</pattern>
> > <shadedPattern>org.apache.flink.shaded.com.google.guava</shadedPattern>
> > </relocation>
> >
> >
> > 2014-11-14 10:22 GMT+01:00 Stephan Ewen <[hidden email]>:
> >
> > > Hey Sebastian!
> > >
> > > I think there are two approaches:
> > >
> > >   - You can either shade the protobuf Classes as well
> > >
> > >   - or you can exclude the ptotobuf namespace from shading. Have a look
> > at
> > > the shade plugin docs.
> > >
> > > Stephan
> > >  Am 14.11.2014 09:49 schrieb "Sebastian Kunert" <[hidden email]>:
> > >
> > > > Hi Guys,
> > > >
> > > > I am currently working on integrating Flink with Mesos. Therefore I
> > > > currently reuse the flink-yarn uberjar assembly to build a jar that
> > > > contains all the classes I need. Now I have a dependency on a Google
> > > > Protobuf class. I added protobuf as a dependency to my module and it
> is
> > > > also contained in the uberjar under /com/google/protobuf.
> > > >
> > > > Nevertheless, on execution I get the following error:
> > > > java.lang.ClassNotFoundException:
> > > > org.apache.flink.shaded.com.google.protobuf.ByteString
> > > >
> > > > The location org.apache.flink.shaded... is wrong. I understand this
> is
> > > > related to the maven shading plugin but after some looking into it I
> am
> > > not
> > > > sure how to fix this.
> > > >
> > > > Greetings,
> > > >
> > > > Sebastian
> > > >
> > >
> >
>