[jira] [Created] (FLINK-11302) FlinkS3FileSystem uses an incorrect path for temporary files.

[Shang Yuanchun (Jira)]

Artsem Semianenka created FLINK-11302:
-----------------------------------------

             Summary: FlinkS3FileSystem uses an incorrect path for temporary files.
                 Key: FLINK-11302
                 URL: https://issues.apache.org/jira/browse/FLINK-11302
             Project: Flink
          Issue Type: Bug
          Components: FileSystem
    Affects Versions: 1.7.1, 1.7.0, 1.8.0
            Reporter: Artsem Semianenka
            Assignee: Artsem Semianenka


I'm running the Flink job which stores the parquet files on S3.
For that purpose, I use StreamingFileSink (in Bulk format) which under the hood uses FlinkS3FileSystem from flink-s3-fs-hadoop-1.7.1.jar

When I try to submit it in Yarn cluster I got the following exception:
{code:java}
java.nio.file.NoSuchFileException: /mnt/e1/yarn/nm/usercache/ec2-user/appcache/application_1544473029846_0487,/mnt/e2/yarn/nm/usercache/ec2-user/appcache/application_1544473029846_0487/.tmp_d7a04184-6fed-4c69-9d2f-f950e06bf5f0
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
        at java.nio.file.Files.newOutputStream(Files.java:216)
        at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:95)
        at org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:43)
        at org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174)
        at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271)
        at org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236)
        at org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:85)
        at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.rollPartFile(Bucket.java:221)
        at org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.write(Bucket.java:212)
        at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.onElement(Buckets.java:268)
        at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.invoke(StreamingFileSink.java:370)
        at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
        ....
{code}
As we can see the target path form stacktrace consist of two paths separated by a comma

It happens because in [AbstractS3FileSystemFactory|https://github.com/apache/flink/blob/bf5dc6c8bac73e7f8d54e983be2080cf6ce48a94/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/AbstractS3FileSystemFactory.java#L141] we use the value of [CoreOptions.TMP_DIRS|https://github.com/apache/flink/blob/43a74977de27869e41e97878e821dcf363d81911/flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java#L216] "as is" and we expect that the value will contain only one path. But in reality, it may contain more than one path.

Proposed solution: Split path by comma and "|" and use only the first path.

Bugfix already exist and I will create a pull request in which 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)