[
https://issues.apache.org/jira/browse/FLINK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034389#comment-14034389 ]
Daniel Warneke commented on FLINK-937:
--------------------------------------
As far as I know, queues are rather an operational feature cluster administrators can use to handle different job priorities or isolate different types of workloads. I don’t see how they could help to find a reasonable default degree of parallelism. To the best of my knowledge, the only YARN APIs we can rely on are:
- Min/max size of containers
- Total number of nodes in the cluster (idle or not)
Therefore, I basically see two possibilities for the default behavior:
- Start large: Determine number of nodes and use them all (Robert’s initial suggestions?)
- Start small: Make default degree of parallelism 1 or 2 (default behavior of MapReduce)
I have a slight preference for option 2, but I would choose either one over the “assistant”…
> Change the YARN Client to allocate all cluster resources, if no argument given
> ------------------------------------------------------------------------------
>
> Key: FLINK-937
> URL:
https://issues.apache.org/jira/browse/FLINK-937> Project: Flink
> Issue Type: Improvement
> Components: YARN Client
> Reporter: Robert Metzger
> Assignee: Robert Metzger
> Fix For: 0.6-incubating
>
>
> In order to further improve the user experience, I would like to change the YARN client's behavior to allocate as many cluster resources as possible, if the user does not specify differently.
> The majority of users have exclusive access to the cluster.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Locked
