Flink limitations under Beam

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink limitations under Beam

amir bahmanyari
Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
Thanks+regards
Reply | Threaded
Open this post in threaded view
|

Re: Flink limitations under Beam

Aljoscha Krettek-2
Hi,
I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink.

I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that.

Best,
Aljoscha
> On 4. Apr 2017, at 02:11, amir bahmanyari <[hidden email]> wrote:
>
> Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
> Thanks+regards

Reply | Threaded
Open this post in threaded view
|

Re: Flink limitations under Beam

amir bahmanyari
Thanks Aljoscha,How sever of performance difference are we talking about here? examples of "Iterations/feedback edges" and how they get applied?Thanks so much Aljoscha.


      From: Aljoscha Krettek <[hidden email]>
 To: [hidden email]; amir bahmanyari <[hidden email]>
 Sent: Tuesday, April 4, 2017 1:59 AM
 Subject: Re: Flink limitations under Beam
   
Hi,
I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink.

I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that.

Best,
Aljoscha
> On 4. Apr 2017, at 02:11, amir bahmanyari <[hidden email]> wrote:
>
> Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
> Thanks+regards


   
Reply | Threaded
Open this post in threaded view
|

Re: Flink limitations under Beam

Aljoscha Krettek-2
Hi,
I don’t have any hard numbers for this, and it depends on what kind of state backend is used (Heap backend or RocksDB backend).

Iterations are documented in the Flink doc: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#iterations <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#iterations> Essentially what they allow is to have a processing graph that is not a DAG. This can be useful in machine learning use cases where you want to feed back some information to an earlier operation to update a model.

Best,
Aljoscha

> On 4. Apr 2017, at 09:46, amir bahmanyari <[hidden email]> wrote:
>
> Thanks Aljoscha,
> How sever of performance difference are we talking about here?
>  examples of "Iterations/feedback edges" and how they get applied?
> Thanks so much Aljoscha.
>
>
>
> From: Aljoscha Krettek <[hidden email]>
> To: [hidden email]; amir bahmanyari <[hidden email]>
> Sent: Tuesday, April 4, 2017 1:59 AM
> Subject: Re: Flink limitations under Beam
>
> Hi,
> I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink.
>
> I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that.
>
> Best,
> Aljoscha
>
> > On 4. Apr 2017, at 02:11, amir bahmanyari <[hidden email] <mailto:[hidden email]>> wrote:
> >
> > Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
> > Thanks+regards
>
>