Posted by
Shang Yuanchun (Jira) on
Jun 17, 2014; 9:23pm
URL: http://deprecated-apache-flink-mailing-list-archive.368.s1.nabble.com/jira-Commented-FLINK-937-Change-the-YARN-Client-to-allocate-all-cluster-resources-if-no-argument-givn-tp359.html
[
https://issues.apache.org/jira/browse/FLINK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034389#comment-14034389 ]
Daniel Warneke commented on FLINK-937:
--------------------------------------
As far as I know, queues are rather an operational feature cluster administrators can use to handle different job priorities or isolate different types of workloads. I don’t see how they could help to find a reasonable default degree of parallelism. To the best of my knowledge, the only YARN APIs we can rely on are:
- Min/max size of containers
- Total number of nodes in the cluster (idle or not)
Therefore, I basically see two possibilities for the default behavior:
- Start large: Determine number of nodes and use them all (Robert’s initial suggestions?)
- Start small: Make default degree of parallelism 1 or 2 (default behavior of MapReduce)
I have a slight preference for option 2, but I would choose either one over the “assistant”…
> Change the YARN Client to allocate all cluster resources, if no argument given
> ------------------------------------------------------------------------------
>
> Key: FLINK-937
> URL:
https://issues.apache.org/jira/browse/FLINK-937> Project: Flink
> Issue Type: Improvement
> Components: YARN Client
> Reporter: Robert Metzger
> Assignee: Robert Metzger
> Fix For: 0.6-incubating
>
>
> In order to further improve the user experience, I would like to change the YARN client's behavior to allocate as many cluster resources as possible, if the user does not specify differently.
> The majority of users have exclusive access to the cluster.
--
This message was sent by Atlassian JIRA
(v6.2#6252)