Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
Thanks+regards |
Hi,
I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink. I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that. Best, Aljoscha > On 4. Apr 2017, at 02:11, amir bahmanyari <[hidden email]> wrote: > > Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner? > Thanks+regards |
Thanks Aljoscha,How sever of performance difference are we talking about here? examples of "Iterations/feedback edges" and how they get applied?Thanks so much Aljoscha.
From: Aljoscha Krettek <[hidden email]> To: [hidden email]; amir bahmanyari <[hidden email]> Sent: Tuesday, April 4, 2017 1:59 AM Subject: Re: Flink limitations under Beam Hi, I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink. I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that. Best, Aljoscha > On 4. Apr 2017, at 02:11, amir bahmanyari <[hidden email]> wrote: > > Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner? > Thanks+regards |
Hi,
I don’t have any hard numbers for this, and it depends on what kind of state backend is used (Heap backend or RocksDB backend). Iterations are documented in the Flink doc: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#iterations <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#iterations> Essentially what they allow is to have a processing graph that is not a DAG. This can be useful in machine learning use cases where you want to feed back some information to an earlier operation to update a model. Best, Aljoscha > On 4. Apr 2017, at 09:46, amir bahmanyari <[hidden email]> wrote: > > Thanks Aljoscha, > How sever of performance difference are we talking about here? > examples of "Iterations/feedback edges" and how they get applied? > Thanks so much Aljoscha. > > > > From: Aljoscha Krettek <[hidden email]> > To: [hidden email]; amir bahmanyari <[hidden email]> > Sent: Tuesday, April 4, 2017 1:59 AM > Subject: Re: Flink limitations under Beam > > Hi, > I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink. > > I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that. > > Best, > Aljoscha > > > On 4. Apr 2017, at 02:11, amir bahmanyari <[hidden email] <mailto:[hidden email]>> wrote: > > > > Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner? > > Thanks+regards > > |
Free forum by Nabble | Edit this page |