We want to change the name of the file being generated as the output of our
StreamFileSink. , when files are generated they are named part-00*, is there a way that we can change the name. In Hadoop, we can change RecordWriters and MultipleOutputs. May I please some help in this regard. This is causing blockers for us and will force us t move to MR job -- Thank you and regards, Dhurandar |
Hi Dhurandar:
Currently StreamingFileSink should be able to change the prefix and suffix of the filename[1], it could be changed to something like <prefix>-0-0<suffix>. Could this solve your problem ? Best, Yun [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html#part-file-configuration ------------------------------------------------------------------ 发件人:dhurandar S<[hidden email]> 日 期:2020年05月13日 05:13:04 收件人:user<[hidden email]>; <[hidden email]> 主 题:changing the output files names in Streamfilesink from part-00 to something else We want to change the name of the file being generated as the output of our StreamFileSink. , when files are generated they are named part-00*, is there a way that we can change the name. In Hadoop, we can change RecordWriters and MultipleOutputs. May I please some help in this regard. This is causing blockers for us and will force us t move to MR job -- Thank you and regards, Dhurandar |
Yes we looked at it ,
The problem is the file name gets generated in a dynamic fashion, based on which organization data we are getting we generate the file name from the coming data. Is there any way we can achieve this ?? On Tue, May 12, 2020 at 8:38 PM Yun Gao <[hidden email]> wrote: > Hi Dhurandar: > > Currently StreamingFileSink should be able to change the prefix and > suffix of the filename[1], it could be changed to something like > <prefix>-0-0<suffix>. Could this solve your problem ? > > > Best, > Yun > > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html#part-file-configuration > > > > ------------------------------------------------------------------ > 发件人:dhurandar S<[hidden email]> > 日 期:2020年05月13日 05:13:04 > 收件人:user<[hidden email]>; <[hidden email]> > 主 题:changing the output files names in Streamfilesink from part-00 to > something else > > We want to change the name of the file being generated as the output of > our StreamFileSink. > , when files are generated they are named part-00*, is there a way that we > can change the name. > > In Hadoop, we can change RecordWriters and MultipleOutputs. May I please > some help in this regard. This is causing blockers for us and will force us > t move to MR job > > -- > Thank you and regards, > Dhurandar > > > -- Thank you and regards, Dhurandar |
The StreamingFileSink can have a OutputFileConfig [1] to config the prefix and suffix of the part file, does that work for you ?
[1] https://github.com/apache/flink/blob/1d9d0bf582a79ed5cba4ec096e9c12fe5618bcf7/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java#L71 Best, Danny Chan 在 2020年5月14日 +0800 AM2:05,dhurandar S <[hidden email]>,写道: > > StreamFileSink |
Free forum by Nabble | Edit this page |