Hi Folks:
I am new to Flink and wanted to find out if there is a guide to how to create new sources/sinks for Flink. I tried browsing through the documentation but did not find a single place with all the interfaces/etc required. Thanks for your help. Mans |
Hi Mans,
there is no explicit guide for how to write a custom source/sink as far as I know. There is just a paragraph in the data sources section ( https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources ). For sources you have to implement the SourceFunction interface for non parallel sources or the ParallelSourceFunction for parallel sources. If you need access to the RuntimeContext, then you should extend the RichParallelSourceFunction. Be aware, that you should always emit elements to the downstream operator when you hold the checkpoint lock. You emit and obtain the checkpoint lock from the SourceContext. For custom sinks you have to implement the SinkFunction or the ParallelSinkFunction. I hope this gets you started :-) Cheers, Till On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> wrote: > Hi Folks: > I am new to Flink and wanted to find out if there is a guide to how to > create new sources/sinks for Flink. > I tried browsing through the documentation but did not find a single place > with all the interfaces/etc required. > Thanks for your help. > Mans |
Thanks Till for the pointers.
Mans On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <[hidden email]> wrote: Hi Mans,there is no explicit guide for how to write a custom source/sink as far as I know. There is just a paragraph in the data sources section (https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For sources you have to implement the SourceFunction interface for non parallel sources or the ParallelSourceFunction for parallel sources. If you need access to the RuntimeContext, then you should extend the RichParallelSourceFunction. Be aware, that you should always emit elements to the downstream operator when you hold the checkpoint lock. You emit and obtain the checkpoint lock from the SourceContext.For custom sinks you have to implement the SinkFunction or the ParallelSinkFunction.I hope this gets you started :-)Cheers, Till On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> wrote: Hi Folks: I am new to Flink and wanted to find out if there is a guide to how to create new sources/sinks for Flink. I tried browsing through the documentation but did not find a single place with all the interfaces/etc required. Thanks for your help. Mans |
Hi Till, Folks:
I found ParallelSourceFunction/RichParallelSourceFunctions in the flink-streaming-java_2.10 module (1.1-SNAPSHOT), but could not find ParallelSinkFunction et. al. Am I missing anything ? Thanks On Tuesday, May 10, 2016 5:31 AM, M Singh <[hidden email]> wrote: Thanks Till for the pointers. Mans On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <[hidden email]> wrote: Hi Mans,there is no explicit guide for how to write a custom source/sink as far as I know. There is just a paragraph in the data sources section (https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For sources you have to implement the SourceFunction interface for non parallel sources or the ParallelSourceFunction for parallel sources. If you need access to the RuntimeContext, then you should extend the RichParallelSourceFunction. Be aware, that you should always emit elements to the downstream operator when you hold the checkpoint lock. You emit and obtain the checkpoint lock from the SourceContext.For custom sinks you have to implement the SinkFunction or the ParallelSinkFunction.I hope this gets you started :-)Cheers, Till On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> wrote: Hi Folks: I am new to Flink and wanted to find out if there is a guide to how to create new sources/sinks for Flink. I tried browsing through the documentation but did not find a single place with all the interfaces/etc required. Thanks for your help. Mans |
I think Till mistyped, sinks are always parallel, you don't need the
ParallelSinkFunction. On Tue, 10 May 2016 at 18:56 M Singh <[hidden email]> wrote: > Hi Till, Folks: > I found ParallelSourceFunction/RichParallelSourceFunctions in the > flink-streaming-java_2.10 module (1.1-SNAPSHOT), but could not find > ParallelSinkFunction et. al. Am I missing anything ? > Thanks > > > > On Tuesday, May 10, 2016 5:31 AM, M Singh <[hidden email]> > wrote: > > > Thanks Till for the pointers. > Mans > > On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Mans,there is no explicit guide for how to write a custom source/sink > as far as I know. There is just a paragraph in the data sources section ( > https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For > sources you have to implement the SourceFunction interface for non parallel > sources or the ParallelSourceFunction for parallel sources. If you need > access to the RuntimeContext, then you should extend the > RichParallelSourceFunction. Be aware, that you should always emit elements > to the downstream operator when you hold the checkpoint lock. You emit and > obtain the checkpoint lock from the SourceContext.For custom sinks you have > to implement the SinkFunction or the ParallelSinkFunction.I hope this gets > you started :-)Cheers, > Till > On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> > wrote: > > Hi Folks: > I am new to Flink and wanted to find out if there is a guide to how to > create new sources/sinks for Flink. > I tried browsing through the documentation but did not find a single place > with all the interfaces/etc required. > Thanks for your help. > Mans > > > > > > |
Yes I did. Sorry my bad.
On May 11, 2016 7:31 AM, "Aljoscha Krettek" <[hidden email]> wrote: > I think Till mistyped, sinks are always parallel, you don't need the > ParallelSinkFunction. > > On Tue, 10 May 2016 at 18:56 M Singh <[hidden email]> wrote: > > > Hi Till, Folks: > > I found ParallelSourceFunction/RichParallelSourceFunctions in the > > flink-streaming-java_2.10 module (1.1-SNAPSHOT), but could not find > > ParallelSinkFunction et. al. Am I missing anything ? > > Thanks > > > > > > > > On Tuesday, May 10, 2016 5:31 AM, M Singh > <[hidden email]> > > wrote: > > > > > > Thanks Till for the pointers. > > Mans > > > > On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann < > [hidden email]> > > wrote: > > > > > > Hi Mans,there is no explicit guide for how to write a custom source/sink > > as far as I know. There is just a paragraph in the data sources section ( > > > https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For > > sources you have to implement the SourceFunction interface for non > parallel > > sources or the ParallelSourceFunction for parallel sources. If you need > > access to the RuntimeContext, then you should extend the > > RichParallelSourceFunction. Be aware, that you should always emit > elements > > to the downstream operator when you hold the checkpoint lock. You emit > and > > obtain the checkpoint lock from the SourceContext.For custom sinks you > have > > to implement the SinkFunction or the ParallelSinkFunction.I hope this > gets > > you started :-)Cheers, > > Till > > On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> > > wrote: > > > > Hi Folks: > > I am new to Flink and wanted to find out if there is a guide to how to > > create new sources/sinks for Flink. > > I tried browsing through the documentation but did not find a single > place > > with all the interfaces/etc required. > > Thanks for your help. > > Mans > > > > > > > > > > > > > |
Free forum by Nabble | Edit this page |