Forwarding Strategies

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Forwarding Strategies

Greg Hogan
Looking at org.apache.flink.runtime.operators.shipping.OutputEmitter,
shipping strategies FORWARD, PARTITION_RANDOM, and
PARTITION_FORCED_REBALANCE all call a local round-robin partitioning
function. I'd like to patch this so that the round-robin count starts at
the local task index, but shouldn't the FORWARD strategy be to always
partition to the local task index ("forwarding the data locally in memory")?

https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/shipping/OutputEmitter.java

Greg
Reply | Threaded
Open this post in threaded view
|

Re: Forwarding Strategies

Stephan Ewen
Hi!

I am not sure what PARTITION_RANDOM is about, but FORWARD and
PARTITION_FORCED_REBALANCE
are the same to the output emitter, but look different to the
Optimizer (PARTITION_FORCED_REBALANCE
cannot be pushed down).

Stephan


On Tue, Oct 20, 2015 at 6:39 PM, Greg Hogan <[hidden email]> wrote:

> Looking at org.apache.flink.runtime.operators.shipping.OutputEmitter,
> shipping strategies FORWARD, PARTITION_RANDOM, and
> PARTITION_FORCED_REBALANCE all call a local round-robin partitioning
> function. I'd like to patch this so that the round-robin count starts at
> the local task index, but shouldn't the FORWARD strategy be to always
> partition to the local task index ("forwarding the data locally in
> memory")?
>
>
> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/shipping/OutputEmitter.java
>
> Greg
>