Which is the right degree of parallism?

[Timo Walther]

Hey everyone,

I want to get the maximum performance of my small 2 node cluster. At the
moment my execution plan has a "parallelism" of "1" at each operator.
What "-p XX" argument should I pass to the job? The number of nodes,
number of CPUs or number of slots?

Thanks and regards,
Timo

[Stephan Ewen]

Hey!

That depends on the job, but in general, #cores is a good point to start.

Stephan



On Tue, Jul 22, 2014 at 2:05 PM, Timo Walther <[hidden email]> wrote:

> Hey everyone,
>
> I want to get the maximum performance of my small 2 node cluster. At the
> moment my execution plan has a "parallelism" of "1" at each operator.
> What "-p XX" argument should I pass to the job? The number of nodes,
> number of CPUs or number of slots?
>
> Thanks and regards,
> Timo
>

[Aljoscha Krettek-2]

Do the slots correlate with the number of cores? I think the slots business
might be confusing for some users.


On Tue, Jul 22, 2014 at 2:31 PM, Stephan Ewen <[hidden email]> wrote:

> Hey!
>
> That depends on the job, but in general, #cores is a good point to start.
>
> Stephan
>
>
>
> On Tue, Jul 22, 2014 at 2:05 PM, Timo Walther <[hidden email]> wrote:
>
> > Hey everyone,
> >
> > I want to get the maximum performance of my small 2 node cluster. At the
> > moment my execution plan has a "parallelism" of "1" at each operator.
> > What "-p XX" argument should I pass to the job? The number of nodes,
> > number of CPUs or number of slots?
> >
> > Thanks and regards,
> > Timo
> >
>

[Ufuk Celebi]

On 22 Jul 2014, at 14:58, Aljoscha Krettek <[hidden email]> wrote:

> Do the slots correlate with the number of cores? I think the slots business
> might be confusing for some users.

I think it depends as well. Number of cores would be a reasonable default for the slots though.

[Stephan Ewen]

The number of slots that a machine offers is what you define in the config.
Setting it to #cores is in many cases reasonable.

Have a look at the default config under "/conf", it has an entry there
where you set the slots per machine.