[jira] [Created] (FLINK-11378) Allow HadoopRecoverableWriter to write to Hadoop compatible Filesystems.

[Shang Yuanchun (Jira)]

Martijn created FLINK-11378:
-------------------------------

             Summary: Allow HadoopRecoverableWriter to write to Hadoop compatible Filesystems.
                 Key: FLINK-11378
                 URL: https://issues.apache.org/jira/browse/FLINK-11378
             Project: Flink
          Issue Type: Improvement
          Components: FileSystem
            Reporter: Martijn


At a client we're using Flink jobs to read data from Kafka and writing it to GCS. In earlier versions, we've used `BucketingFileSink` for this, but we want to switch to the newer `StreamingFileSink`.

Since we're running Flink on Google's DataProc, we're using the Hadoop compatible GCS [connector|https://github.com/GoogleCloudPlatform/bigdata-interop] made by Google. This currently doesn't work on Flink, because Flink checks for a HDFS scheme at 'HadoopRecoverableWriter'.

We've successfully ran our jobs by creating a custom Flink Distro which has the hdfs scheme check removed.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)