[jira] [Created] (FLINK-18181) StreamingFileCommitter should not use fs modification time for proc committer

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (FLINK-18181) StreamingFileCommitter should not use fs modification time for proc committer

Shang Yuanchun (Jira)
Jingsong Lee created FLINK-18181:
------------------------------------

             Summary: StreamingFileCommitter should not use fs modification time for proc committer
                 Key: FLINK-18181
                 URL: https://issues.apache.org/jira/browse/FLINK-18181
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem
            Reporter: Jingsong Lee
            Assignee: Jingsong Lee
             Fix For: 1.11.0


The filesystem path modification time is the last modification time of all files in directory. 

If there is new data in partition all the time, it will never end.

We should use earlier time for committer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)