xiaojin.wy created FLINK-15098:
----------------------------------
Summary: CsvTableSink has no properties of a OVERWRITE mode to write, but the exception hint the user use it when a file or directory already exists
Key: FLINK-15098
URL:
https://issues.apache.org/jira/browse/FLINK-15098 Project: Flink
Issue Type: Improvement
Components: Table SQL / Client
Affects Versions: 1.10.0
Reporter: xiaojin.wy
When using sql-client to execute a sql , there will be a IOException just like the content below if the file or directory already exists. But there is no place for the user to set the overwrite mode. In another word, the hint tell users to use overwrite mode, but flink has not support it now.
The exception:
Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, count$1) AS avg_1]) -> SinkConversionToRow -> Map -> Sink: CsvTableSink(avg_1)': File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories.Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'HashAggregate(isMerge=[true], select=[Final_AVG(sum$0, count$1) AS avg_1]) -> SinkConversionToRow -> Map -> Sink: CsvTableSink(avg_1)': File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories. at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216) at org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:243) at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:215) at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:203) at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:119) at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:104) at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278) at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:266) at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:146) ... 10 moreCaused by: java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories. at org.apache.flink.core.fs.FileSystem.initOutPathDistFS(FileSystem.java:937) at org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:286) at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:100) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:212) ... 20 more
The properties that a csvTableSink support now:
connector.type, connector.property-version, connector.path, format.type, format.property-version, format.field-delimiter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
