Translate one Java API operator to multiple common API operators?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Translate one Java API operator to multiple common API operators?

Viktor Rosenfeld
Hi,

Is it possible to implement one Java API operator using multiple Common API operators?

I.e., turn something like

   IN -> OP(aggregate) -> OUT

into

   IN -> Map1 -> Reduce -> Map2 -> OUT

I would imagine that the method SingleInputOperator.translateToDataFlow could just create a chain of operators, wire the input to the first operator (Map1) and return the last operator of the chain (Map2).

Is there anything I need to be aware of with this approach, e.g., some performance issues?

Best,
Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Translate one Java API operator to multiple common API operators?

Fabian Hueske
This is possible and already done.
For example, a key-selector-groupReduce is translated into sth like:
map(keySelector)->groupReduce()->map(unwrap)

Since, MapFunctions are quite lightweight, there shouldn't be major
performance implications. Make sure you get the DOP of all operators right,
though.

2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <[hidden email]>:

> Hi,
>
> Is it possible to implement one Java API operator using multiple Common API
> operators?
>
> I.e., turn something like
>
>    IN -> OP(aggregate) -> OUT
>
> into
>
>    IN -> Map1 -> Reduce -> Map2 -> OUT
>
> I would imagine that the method SingleInputOperator.translateToDataFlow
> could just create a chain of operators, wire the input to the first
> operator
> (Map1) and return the last operator of the chain (Map2).
>
> Is there anything I need to be aware of with this approach, e.g., some
> performance issues?
>
> Best,
> Viktor
>
>
>
> --
> View this message in context:
> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
> archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Translate one Java API operator to multiple common API operators?

Fabian Hueske
correction, there is no map(unwrap) in the example I gave.

2014-11-21 13:15 GMT+01:00 Fabian Hueske <[hidden email]>:

> This is possible and already done.
> For example, a key-selector-groupReduce is translated into sth like:
> map(keySelector)->groupReduce()->map(unwrap)
>
> Since, MapFunctions are quite lightweight, there shouldn't be major
> performance implications. Make sure you get the DOP of all operators right,
> though.
>
> 2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <[hidden email]
> >:
>
>> Hi,
>>
>> Is it possible to implement one Java API operator using multiple Common
>> API
>> operators?
>>
>> I.e., turn something like
>>
>>    IN -> OP(aggregate) -> OUT
>>
>> into
>>
>>    IN -> Map1 -> Reduce -> Map2 -> OUT
>>
>> I would imagine that the method SingleInputOperator.translateToDataFlow
>> could just create a chain of operators, wire the input to the first
>> operator
>> (Map1) and return the last operator of the chain (Map2).
>>
>> Is there anything I need to be aware of with this approach, e.g., some
>> performance issues?
>>
>> Best,
>> Viktor
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
>> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>