Custom Partitioners Syntax

[Stephan Ewen]

Hi all!

Custom partitioners allow you manually define the assignment of keys to
partitions, for cases that have special constraints.

This is a call for opinion on the Syntax for custom partitioners, in the
case of Join and CoGroup.

Option 1:

input1
    .join(input2)
    .where("key1").equalTo("key2")

    .withPartitioner( (key, num) -> key % num )

    .with( (right, left) -> new Tuple3<>(right.key1, right.name,
left.sizeOfEyebrow) )

Option 2:

input1
    .join(input2)
    .where("key1").equalTo("key2")
    .with( (right, left) -> new Tuple3<>(right.key1, right.name,
left.sizeOfEyebrow) )

    .withPartitioner( (key, num) -> key % num )


(The UDF and the partitioner calls are swapped here) (CoGroup would be
analogous)

Greetings,
Stephan

[Fabian Hueske]

I prefer the first option where partitioning (assigning keys to partitions)
follows key selection.


2014-11-24 19:52 GMT+01:00 Stephan Ewen <[hidden email]>:

> Hi all!
>
> Custom partitioners allow you manually define the assignment of keys to
> partitions, for cases that have special constraints.
>
> This is a call for opinion on the Syntax for custom partitioners, in the
> case of Join and CoGroup.
>
> Option 1:
>
> input1
>     .join(input2)
>     .where("key1").equalTo("key2")
>
>     .withPartitioner( (key, num) -> key % num )
>
>     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> left.sizeOfEyebrow) )
>
> Option 2:
>
> input1
>     .join(input2)
>     .where("key1").equalTo("key2")
>     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> left.sizeOfEyebrow) )
>
>     .withPartitioner( (key, num) -> key % num )
>
>
> (The UDF and the partitioner calls are swapped here) (CoGroup would be
> analogous)
>
> Greetings,
> Stephan
>

[Stephan Ewen]

I actually prefer the first one as well...
Am 24.11.2014 20:56 schrieb "Fabian Hueske" <[hidden email]>:

> I prefer the first option where partitioning (assigning keys to partitions)
> follows key selection.
>
>
> 2014-11-24 19:52 GMT+01:00 Stephan Ewen <[hidden email]>:
>
> > Hi all!
> >
> > Custom partitioners allow you manually define the assignment of keys to
> > partitions, for cases that have special constraints.
> >
> > This is a call for opinion on the Syntax for custom partitioners, in the
> > case of Join and CoGroup.
> >
> > Option 1:
> >
> > input1
> >     .join(input2)
> >     .where("key1").equalTo("key2")
> >
> >     .withPartitioner( (key, num) -> key % num )
> >
> >     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> > left.sizeOfEyebrow) )
> >
> > Option 2:
> >
> > input1
> >     .join(input2)
> >     .where("key1").equalTo("key2")
> >     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> > left.sizeOfEyebrow) )
> >
> >     .withPartitioner( (key, num) -> key % num )
> >
> >
> > (The UDF and the partitioner calls are swapped here) (CoGroup would be
> > analogous)
> >
> > Greetings,
> > Stephan
> >
>

[till.rohrmann]

The latter version would allow to use the apply method in Scala
without calling it directly, whereas in the first case the user would
have to spell it out.

On Mon, Nov 24, 2014 at 8:56 PM, Fabian Hueske <[hidden email]> wrote:

> I prefer the first option where partitioning (assigning keys to partitions)
> follows key selection.
>
>
> 2014-11-24 19:52 GMT+01:00 Stephan Ewen <[hidden email]>:
>
>> Hi all!
>>
>> Custom partitioners allow you manually define the assignment of keys to
>> partitions, for cases that have special constraints.
>>
>> This is a call for opinion on the Syntax for custom partitioners, in the
>> case of Join and CoGroup.
>>
>> Option 1:
>>
>> input1
>>     .join(input2)
>>     .where("key1").equalTo("key2")
>>
>>     .withPartitioner( (key, num) -> key % num )
>>
>>     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
>> left.sizeOfEyebrow) )
>>
>> Option 2:
>>
>> input1
>>     .join(input2)
>>     .where("key1").equalTo("key2")
>>     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
>> left.sizeOfEyebrow) )
>>
>>     .withPartitioner( (key, num) -> key % num )
>>
>>
>> (The UDF and the partitioner calls are swapped here) (CoGroup would be
>> analogous)
>>
>> Greetings,
>> Stephan
>>

[Stephan Ewen]

Ah, that is actually a good argument...

On Mon, Nov 24, 2014 at 11:50 PM, Till Rohrmann <[hidden email]>
wrote:

> The latter version would allow to use the apply method in Scala
> without calling it directly, whereas in the first case the user would
> have to spell it out.
>
> On Mon, Nov 24, 2014 at 8:56 PM, Fabian Hueske <[hidden email]> wrote:
> > I prefer the first option where partitioning (assigning keys to
> partitions)
> > follows key selection.
> >
> >
> > 2014-11-24 19:52 GMT+01:00 Stephan Ewen <[hidden email]>:
> >
> >> Hi all!
> >>
> >> Custom partitioners allow you manually define the assignment of keys to
> >> partitions, for cases that have special constraints.
> >>
> >> This is a call for opinion on the Syntax for custom partitioners, in the
> >> case of Join and CoGroup.
> >>
> >> Option 1:
> >>
> >> input1
> >>     .join(input2)
> >>     .where("key1").equalTo("key2")
> >>
> >>     .withPartitioner( (key, num) -> key % num )
> >>
> >>     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> >> left.sizeOfEyebrow) )
> >>
> >> Option 2:
> >>
> >> input1
> >>     .join(input2)
> >>     .where("key1").equalTo("key2")
> >>     .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> >> left.sizeOfEyebrow) )
> >>
> >>     .withPartitioner( (key, num) -> key % num )
> >>
> >>
> >> (The UDF and the partitioner calls are swapped here) (CoGroup would be
> >> analogous)
> >>
> >> Greetings,
> >> Stephan
> >>
>