Israel Ekpo created FLINK-17989:
-----------------------------------
Summary: java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
Key: FLINK-17989
URL:
https://issues.apache.org/jira/browse/FLINK-17989 Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem, FileSystems
Affects Versions: 1.10.1, 1.9.3
Environment: Ubuntu 18
Java 1.8
Flink 1.9.x and 1.10.x
Reporter: Israel Ekpo
Fix For: 1.10.1, 1.9.3
In the POM.xml classes from certain packages are relocated and filtered out of the final jar In the POM.xml classes from certain packages are relocated and filtered out of the final jar
This is causing errors for customers and users that are using the StreamingFileSink with Azure Blob Storage in Flink version 1.9.x, 1.10.x and possibly 1.11.x
<a href="https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233">https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233
I would like to know why the relocation is happening and the reasoning behind the exclusion and filtering of the classes
It seems to affect just the Azure file systems in my sample implementations
{code:java}
String outputPath = "wasbs://
[hidden email]/streaming-output/";
final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
stream.addSink(sink);
// execute programenv.execute(StreamingJob.class.getCanonicalName());
{code}
2020-05-27 17:23:16java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.ClassNotFoundException: org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60) at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
