Tony Xintong Song created FLINK-11166:
-----------------------------------------
Summary: Slot Placement Constraint
Key: FLINK-11166
URL:
https://issues.apache.org/jira/browse/FLINK-11166 Project: Flink
Issue Type: New Feature
Components: ResourceManager
Reporter: Tony Xintong Song
In many cases, users may want Flink to schedule their job tasks following certain locality preferences. E.g., colocating upstream/downstream tasks to reduce data transmission costs, dispersing tasks of certain pattern (e.g., I/O intensive) to avoid resource competitions, running tasks in exclusive TaskExecutor-s for task level resource consumption measurements, etc.
Currently, there are two ways in Flink to specify such locality preferences: specifying preferred locations in the slot request, or setting slot sharing group for the task. In both ways the preferences are specified when requesting slots from the SlotPool and can affect how tasks are placed among the slots allocated to the JobMaster.
However, there is no guarantee that such preferences can always be satisfied, especially when slots are customized with different resource profiles for different kinds of tasks. E.g., in cases where two tasks A and B are preferred to be scheduled onto a same TaskExecutor, it is possible that none of the slots customized for A offered to the JobMaster are collocated with slots customized for B.
To better support locality preferences with various slot resource specifications, it is necessary to allow JobMaster-s to request slots subjected to certain placement constraints from the ResourceManager.
In addition, most underlying frameworks Flink runs on (Yarn, Kubernetes, Mesos) already have individual supports for container level placement constraints. It is a great opportunity for Flink to leverage such underlying supports and enable scheduling tasks with rich locality preferences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
