Flink S3 Hadoop dependencies

[Satish Saley]

Hi team,

Was there a reason for not shading hadoop-common
https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44
? This is leaking lots of classes such as guava and causing issues in our
flink application.
I see that hadoop-common classes were shaded in earlier versions
https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0

Stacktrace :
Caused by: java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
at io.grpc.Metadata$Key.validateName(Metadata.java:742)
at io.grpc.Metadata$Key.<init>(Metadata.java:750)
at io.grpc.Metadata$Key.<init>(Metadata.java:668)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
at io.grpc.Metadata$Key.of(Metadata.java:705)
at io.grpc.Metadata$Key.of(Metadata.java:701)
at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
at
io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
at
io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
at
io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)

[Chesnay Schepler-3]

Filesystems are supposed to be used as plugins (by putting the jars
under plugins/ instead of lib/), in which case they are loaded
separately from other classes, specifically user-code.

https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/plugins.html

On 14/08/2020 20:25, Satish Saley wrote:

> Hi team,
>
> Was there a reason for not shading hadoop-common
> https://github.com/apache/flink/commit/e1e7d7f7ecc080c850a264021bf1b20e3d27d373#diff-e7b798a682ee84ab804988165e99761cR38-R44 
> ? This is leaking lots of classes such as guava and causing issues in
> our flink application.
> I see that hadoop-common classes were shaded in earlier versions
> https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop/1.9.0
>
> Stacktrace :
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
> at io.grpc.Metadata$Key.validateName(Metadata.java:742)
> at io.grpc.Metadata$Key.<init>(Metadata.java:750)
> at io.grpc.Metadata$Key.<init>(Metadata.java:668)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
> at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
> at io.grpc.Metadata$Key.of(Metadata.java:705)
> at io.grpc.Metadata$Key.of(Metadata.java:701)
> at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
> at
> io.grpc.internal.AbstractManagedChannelImplBuilder.<clinit>(AbstractManagedChannelImplBuilder.java:90)
> at
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:42)
> at
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForTarget(NettyChannelProvider.java:23)
> at io.grpc.ManagedChannelBuilder.forTarget(ManagedChannelBuilder.java:76)