chaganti spurthi created FLINK-17253:
----------------------------------------
Summary: Support writing to viewfs for hadoop versions < 2.7 when using BulkFormatBuilder in StreamingFileSink
Key: FLINK-17253
URL:
https://issues.apache.org/jira/browse/FLINK-17253 Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.9.0, 1.8.3, 1.8.2, 1.8.1, 1.8.0
Reporter: chaganti spurthi
FLINK-14170 introduced Hadoop version check to support older hadoop versions. However the check only included "hdfs" scheme but not "viewfs". We are using StreamingFileSink to write data to our federated hadoop cluster with cdh-2.6 hadoop version and we are hit with
{code:java}
java.lang.UnsupportedOperationException: Recoverable writers on Hadoop are only supported for HDFS and for Hadoop version 2.7 or newer at org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:61) at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:317) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748)
{code}
The change is add viewfs to the scheme check.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
