Guide for Flink Source/Sinks

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Guide for Flink Source/Sinks

M Singh
Hi Folks:
I am new to Flink and wanted to find out if there is a guide to how to create new sources/sinks for Flink.
I tried browsing through the documentation but did not find a single place with all the interfaces/etc required.
Thanks for your help.
Mans
Reply | Threaded
Open this post in threaded view
|

Re: Guide for Flink Source/Sinks

Till Rohrmann
Hi Mans,

there is no explicit guide for how to write a custom source/sink as far as
I know. There is just a paragraph in the data sources section (
https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources
).

For sources you have to implement the SourceFunction interface for non
parallel sources or the ParallelSourceFunction for parallel sources. If you
need access to the RuntimeContext, then you should extend the
RichParallelSourceFunction. Be aware, that you should always emit elements
to the downstream operator when you hold the checkpoint lock. You emit and
obtain the checkpoint lock from the SourceContext.

For custom sinks you have to implement the SinkFunction or the
ParallelSinkFunction.

I hope this gets you started :-)

Cheers,
Till


On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]>
wrote:

> Hi Folks:
> I am new to Flink and wanted to find out if there is a guide to how to
> create new sources/sinks for Flink.
> I tried browsing through the documentation but did not find a single place
> with all the interfaces/etc required.
> Thanks for your help.
> Mans
Reply | Threaded
Open this post in threaded view
|

Re: Guide for Flink Source/Sinks

M Singh
Thanks Till for the pointers.
Mans

    On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <[hidden email]> wrote:
 

 Hi Mans,there is no explicit guide for how to write a custom source/sink as far as I know. There is just a paragraph in the data sources section (https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For sources you have to implement the SourceFunction interface for non parallel sources or the ParallelSourceFunction for parallel sources. If you need access to the RuntimeContext, then you should extend the RichParallelSourceFunction. Be aware, that you should always emit elements to the downstream operator when you hold the checkpoint lock. You emit and obtain the checkpoint lock from the SourceContext.For custom sinks you have to implement the SinkFunction or the ParallelSinkFunction.I hope this gets you started :-)Cheers,
Till​
On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> wrote:

Hi Folks:
I am new to Flink and wanted to find out if there is a guide to how to create new sources/sinks for Flink.
I tried browsing through the documentation but did not find a single place with all the interfaces/etc required.
Thanks for your help.
Mans



Reply | Threaded
Open this post in threaded view
|

Re: Guide for Flink Source/Sinks

M Singh
Hi Till, Folks:
I found ParallelSourceFunction/RichParallelSourceFunctions in the flink-streaming-java_2.10 module (1.1-SNAPSHOT), but could not find ParallelSinkFunction et. al.  Am I missing anything ?
Thanks

 

    On Tuesday, May 10, 2016 5:31 AM, M Singh <[hidden email]> wrote:
 

 Thanks Till for the pointers.
Mans

    On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <[hidden email]> wrote:
 

 Hi Mans,there is no explicit guide for how to write a custom source/sink as far as I know. There is just a paragraph in the data sources section (https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For sources you have to implement the SourceFunction interface for non parallel sources or the ParallelSourceFunction for parallel sources. If you need access to the RuntimeContext, then you should extend the RichParallelSourceFunction. Be aware, that you should always emit elements to the downstream operator when you hold the checkpoint lock. You emit and obtain the checkpoint lock from the SourceContext.For custom sinks you have to implement the SinkFunction or the ParallelSinkFunction.I hope this gets you started :-)Cheers,
Till​
On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]> wrote:

Hi Folks:
I am new to Flink and wanted to find out if there is a guide to how to create new sources/sinks for Flink.
I tried browsing through the documentation but did not find a single place with all the interfaces/etc required.
Thanks for your help.
Mans





Reply | Threaded
Open this post in threaded view
|

Re: Guide for Flink Source/Sinks

Aljoscha Krettek-2
I think Till mistyped, sinks are always parallel, you don't need the
ParallelSinkFunction.

On Tue, 10 May 2016 at 18:56 M Singh <[hidden email]> wrote:

> Hi Till, Folks:
> I found ParallelSourceFunction/RichParallelSourceFunctions in the
> flink-streaming-java_2.10 module (1.1-SNAPSHOT), but could not find
> ParallelSinkFunction et. al.  Am I missing anything ?
> Thanks
>
>
>
>     On Tuesday, May 10, 2016 5:31 AM, M Singh <[hidden email]>
> wrote:
>
>
>  Thanks Till for the pointers.
> Mans
>
>     On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <[hidden email]>
> wrote:
>
>
>  Hi Mans,there is no explicit guide for how to write a custom source/sink
> as far as I know. There is just a paragraph in the data sources section (
> https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For
> sources you have to implement the SourceFunction interface for non parallel
> sources or the ParallelSourceFunction for parallel sources. If you need
> access to the RuntimeContext, then you should extend the
> RichParallelSourceFunction. Be aware, that you should always emit elements
> to the downstream operator when you hold the checkpoint lock. You emit and
> obtain the checkpoint lock from the SourceContext.For custom sinks you have
> to implement the SinkFunction or the ParallelSinkFunction.I hope this gets
> you started :-)Cheers,
> Till​
> On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]>
> wrote:
>
> Hi Folks:
> I am new to Flink and wanted to find out if there is a guide to how to
> create new sources/sinks for Flink.
> I tried browsing through the documentation but did not find a single place
> with all the interfaces/etc required.
> Thanks for your help.
> Mans
>
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Guide for Flink Source/Sinks

Till Rohrmann
Yes I did. Sorry my bad.
On May 11, 2016 7:31 AM, "Aljoscha Krettek" <[hidden email]> wrote:

> I think Till mistyped, sinks are always parallel, you don't need the
> ParallelSinkFunction.
>
> On Tue, 10 May 2016 at 18:56 M Singh <[hidden email]> wrote:
>
> > Hi Till, Folks:
> > I found ParallelSourceFunction/RichParallelSourceFunctions in the
> > flink-streaming-java_2.10 module (1.1-SNAPSHOT), but could not find
> > ParallelSinkFunction et. al.  Am I missing anything ?
> > Thanks
> >
> >
> >
> >     On Tuesday, May 10, 2016 5:31 AM, M Singh
> <[hidden email]>
> > wrote:
> >
> >
> >  Thanks Till for the pointers.
> > Mans
> >
> >     On Tuesday, May 10, 2016 1:07 AM, Till Rohrmann <
> [hidden email]>
> > wrote:
> >
> >
> >  Hi Mans,there is no explicit guide for how to write a custom source/sink
> > as far as I know. There is just a paragraph in the data sources section (
> >
> https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/#data-sources).For
> > sources you have to implement the SourceFunction interface for non
> parallel
> > sources or the ParallelSourceFunction for parallel sources. If you need
> > access to the RuntimeContext, then you should extend the
> > RichParallelSourceFunction. Be aware, that you should always emit
> elements
> > to the downstream operator when you hold the checkpoint lock. You emit
> and
> > obtain the checkpoint lock from the SourceContext.For custom sinks you
> have
> > to implement the SinkFunction or the ParallelSinkFunction.I hope this
> gets
> > you started :-)Cheers,
> > Till​
> > On Mon, May 9, 2016 at 11:54 PM, M Singh <[hidden email]>
> > wrote:
> >
> > Hi Folks:
> > I am new to Flink and wanted to find out if there is a guide to how to
> > create new sources/sinks for Flink.
> > I tried browsing through the documentation but did not find a single
> place
> > with all the interfaces/etc required.
> > Thanks for your help.
> > Mans
> >
> >
> >
> >
> >
> >
>