[jira] [Created] (FLINK-18130) File name conflict for different jobs in filesystem/hive sink

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (FLINK-18130) File name conflict for different jobs in filesystem/hive sink

Shang Yuanchun (Jira)
Jingsong Lee created FLINK-18130:
------------------------------------

             Summary: File name conflict for different jobs in filesystem/hive sink
                 Key: FLINK-18130
                 URL: https://issues.apache.org/jira/browse/FLINK-18130
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem, Connectors / Hive
            Reporter: Jingsong Lee
            Assignee: Jingsong Lee
             Fix For: 1.11.0


The sink of different jobs (or the same SQL runs multiple times) will produce the same file name, the rename will fail, and different file systems will produce different results:
 * HadoopFileSystem: ignore new data.
 * LocalFileSystem: overwrite old data.

We can add UUID to prefix, make sure there is no conflict between jobs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)