Zhu Zhu created FLINK-14162:
-------------------------------
Summary: Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies
Key: FLINK-14162
URL:
https://issues.apache.org/jira/browse/FLINK-14162 Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zhu Zhu
Fix For: 1.10.0
In scheduler NG, scheduling strategies invokes {{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}} to trigger scheduling of tasks.
However, {{EagerSchedulingStrategy}} and {{LazyFromSourcesSchedulingStrategy}} both invokes it by passing a batch of tasks, but requires the scheduling process to be conducted in 2 different ways:
* {{EagerSchedulingStrategy}} requires the batch of tasks to deploy after all of them have acquired slots. This is essential to avoid partition update RPCs in streaming job scheduling.
* {{LazyFromSourcesSchedulingStrategy}} requires tasks in the batch to allocate slot and get deployed individually, so that it can deploy a few tasks even if the slots is not enough for all tasks in the batch. This is helpful for batch job scheduling.
The scheduler then have to decide the scheduling pattern based whether the scheduling strategy is a {{LazyFromSourcesSchedulingStrategy}}. This is not good, as there can be more strategies in the future, and even customized scheduling strategies.
I think it's better to define the {{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}} to be that all tasks in the batch need to be assigned and deployed together, like what we do for {{EagerSchedulingStrategy}}.
All scheduling strategies need to follow this rule. If tasks should be scheduled individually, the strategy should invoke {{allocateSlotsAndDeploy}} multiple times, one for each task. As a result, the {{LazyFromSourcesSchedulingStrategy}} needs to be adjusted for that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
