Reduce combiner not chained

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Reduce combiner not chained

Ufuk Celebi-2
Hey all,

on the current master running the WordCount example with a text file input/output results and a manual reduce function (instead of the sum(1)) results in a combiner, which is not chained.

The corresponding issue is here: https://issues.apache.org/jira/browse/FLINK-2246

Can someone please confirm this? If it is an issue, we should fix it soon. The serialization overhead is noticeable on larger inputs.

– Ufuk
Reply | Threaded
Open this post in threaded view
|

Re: Reduce combiner not chained

Fabian Hueske-2
This is not a bug. Chained combiners are not supported for ReduceFunctions
yet. :-(
I updated the JIRA accordingly.

2015-06-19 13:04 GMT+02:00 Ufuk Celebi <[hidden email]>:

> Hey all,
>
> on the current master running the WordCount example with a text file
> input/output results and a manual reduce function (instead of the sum(1))
> results in a combiner, which is not chained.
>
> The corresponding issue is here:
> https://issues.apache.org/jira/browse/FLINK-2246
>
> Can someone please confirm this? If it is an issue, we should fix it soon.
> The serialization overhead is noticeable on larger inputs.
>
> – Ufuk
Reply | Threaded
Open this post in threaded view
|

Re: Reduce combiner not chained

Ufuk Celebi-2
I actually thought that this has been improved (not fixed ;)) some time ago. My mistake. Thanks for the update and checking again.

On 19 Jun 2015, at 13:47, Fabian Hueske <[hidden email]> wrote:

> This is not a bug. Chained combiners are not supported for ReduceFunctions
> yet. :-(
> I updated the JIRA accordingly.
>
> 2015-06-19 13:04 GMT+02:00 Ufuk Celebi <[hidden email]>:
>
>> Hey all,
>>
>> on the current master running the WordCount example with a text file
>> input/output results and a manual reduce function (instead of the sum(1))
>> results in a combiner, which is not chained.
>>
>> The corresponding issue is here:
>> https://issues.apache.org/jira/browse/FLINK-2246
>>
>> Can someone please confirm this? If it is an issue, we should fix it soon.
>> The serialization overhead is noticeable on larger inputs.
>>
>> – Ufuk