Hi,
Is it possible to implement one Java API operator using multiple Common API operators? I.e., turn something like IN -> OP(aggregate) -> OUT into IN -> Map1 -> Reduce -> Map2 -> OUT I would imagine that the method SingleInputOperator.translateToDataFlow could just create a chain of operators, wire the input to the first operator (Map1) and return the last operator of the chain (Map2). Is there anything I need to be aware of with this approach, e.g., some performance issues? Best, Viktor |
This is possible and already done.
For example, a key-selector-groupReduce is translated into sth like: map(keySelector)->groupReduce()->map(unwrap) Since, MapFunctions are quite lightweight, there shouldn't be major performance implications. Make sure you get the DOP of all operators right, though. 2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <[hidden email]>: > Hi, > > Is it possible to implement one Java API operator using multiple Common API > operators? > > I.e., turn something like > > IN -> OP(aggregate) -> OUT > > into > > IN -> Map1 -> Reduce -> Map2 -> OUT > > I would imagine that the method SingleInputOperator.translateToDataFlow > could just create a chain of operators, wire the input to the first > operator > (Map1) and return the last operator of the chain (Map2). > > Is there anything I need to be aware of with this approach, e.g., some > performance issues? > > Best, > Viktor > > > > -- > View this message in context: > http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html > Sent from the Apache Flink (Incubator) Mailing List archive. mailing list > archive at Nabble.com. > |
correction, there is no map(unwrap) in the example I gave.
2014-11-21 13:15 GMT+01:00 Fabian Hueske <[hidden email]>: > This is possible and already done. > For example, a key-selector-groupReduce is translated into sth like: > map(keySelector)->groupReduce()->map(unwrap) > > Since, MapFunctions are quite lightweight, there shouldn't be major > performance implications. Make sure you get the DOP of all operators right, > though. > > 2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <[hidden email] > >: > >> Hi, >> >> Is it possible to implement one Java API operator using multiple Common >> API >> operators? >> >> I.e., turn something like >> >> IN -> OP(aggregate) -> OUT >> >> into >> >> IN -> Map1 -> Reduce -> Map2 -> OUT >> >> I would imagine that the method SingleInputOperator.translateToDataFlow >> could just create a chain of operators, wire the input to the first >> operator >> (Map1) and return the last operator of the chain (Map2). >> >> Is there anything I need to be aware of with this approach, e.g., some >> performance issues? >> >> Best, >> Viktor >> >> >> >> -- >> View this message in context: >> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html >> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list >> archive at Nabble.com. >> > > |
Free forum by Nabble | Edit this page |