[PROGRESS-UPDATE] Redesign Flink Scheduling, introducing dedicated Scheduler Component

[Gary Yao-3]

Hi all,

As you might have seen already, we are currently reworking Flink's
scheduling.
At the moment, scheduling is a concern that is scattered across different
components, such as ExecutionGraph, Execution and SlotPool. Scheduling also
happens only on the granularity of individual tasks, which make holistic
scheduling strategies hard to implement. For more details on our motivation
see [1]. To track the progress, we have created the umbrella issue
FLINK-10429
[2] (bear in mind that the current sub-tasks are still subject to change).

We are currently in the process of finalizing the scheduler interfaces. Our
current state can be found in [3]. Feel free to review and comment on our
design proposal.

Best,
Gary

[1]
https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit
[2] https://issues.apache.org/jira/browse/FLINK-10429
[3]
https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU/edit?usp=sharing

[Till Rohrmann]

Thanks for sharing the current state of the scheduler refactorings with the
community Gary. The proposed changes look good to me and, hence, +1 for
proceeding with bringing the interfaces into place.

Cheers,
Till

On Fri, Apr 12, 2019 at 5:53 PM Gary Yao <[hidden email]> wrote:

> Hi all,
>
> As you might have seen already, we are currently reworking Flink's
> scheduling.
> At the moment, scheduling is a concern that is scattered across different
> components, such as ExecutionGraph, Execution and SlotPool. Scheduling also
> happens only on the granularity of individual tasks, which make holistic
> scheduling strategies hard to implement. For more details on our motivation
> see [1]. To track the progress, we have created the umbrella issue
> FLINK-10429
> [2] (bear in mind that the current sub-tasks are still subject to change).
>
> We are currently in the process of finalizing the scheduler interfaces. Our
> current state can be found in [3]. Feel free to review and comment on our
> design proposal.
>
> Best,
> Gary
>
> [1]
>
> https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit
> [2] https://issues.apache.org/jira/browse/FLINK-10429
> [3]
>
> https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU/edit?usp=sharing
>

[Guowei Ma]

Thanks Gary for sharing the documents to the community.
The idea makes scheduler more flexible.

Best,
Guowei


Till Rohrmann <[hidden email]> 于2019年4月13日周六 上午12:21写道：

> Thanks for sharing the current state of the scheduler refactorings with the
> community Gary. The proposed changes look good to me and, hence, +1 for
> proceeding with bringing the interfaces into place.
>
> Cheers,
> Till
>
> On Fri, Apr 12, 2019 at 5:53 PM Gary Yao <[hidden email]> wrote:
>
> > Hi all,
> >
> > As you might have seen already, we are currently reworking Flink's
> > scheduling.
> > At the moment, scheduling is a concern that is scattered across different
> > components, such as ExecutionGraph, Execution and SlotPool. Scheduling
> also
> > happens only on the granularity of individual tasks, which make holistic
> > scheduling strategies hard to implement. For more details on our
> motivation
> > see [1]. To track the progress, we have created the umbrella issue
> > FLINK-10429
> > [2] (bear in mind that the current sub-tasks are still subject to
> change).
> >
> > We are currently in the process of finalizing the scheduler interfaces.
> Our
> > current state can be found in [3]. Feel free to review and comment on our
> > design proposal.
> >
> > Best,
> > Gary
> >
> > [1]
> >
> >
> https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit
> > [2] https://issues.apache.org/jira/browse/FLINK-10429
> > [3]
> >
> >
> https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU/edit?usp=sharing
> >
>

[Zhijiang(wangzhijiang999)]

Thanks for sharing the latest updates for the scheduler refactoring Gary.
It is really good to me and wish seeing the new interface ready soon.

Best,
Zhijiang
------------------------------------------------------------------
From:Guowei Ma <[hidden email]>
Send Time:2019年4月15日(星期一) 09:25
To:dev <[hidden email]>
Subject:Re: [PROGRESS-UPDATE] Redesign Flink Scheduling, introducing dedicated Scheduler Component

Thanks Gary for sharing the documents to the community.
The idea makes scheduler more flexible.

Best,
Guowei


Till Rohrmann <[hidden email]> 于2019年4月13日周六 上午12:21写道：

> Thanks for sharing the current state of the scheduler refactorings with the
> community Gary. The proposed changes look good to me and, hence, +1 for
> proceeding with bringing the interfaces into place.
>
> Cheers,
> Till
>
> On Fri, Apr 12, 2019 at 5:53 PM Gary Yao <[hidden email]> wrote:
>
> > Hi all,
> >
> > As you might have seen already, we are currently reworking Flink's
> > scheduling.
> > At the moment, scheduling is a concern that is scattered across different
> > components, such as ExecutionGraph, Execution and SlotPool. Scheduling
> also
> > happens only on the granularity of individual tasks, which make holistic
> > scheduling strategies hard to implement. For more details on our
> motivation
> > see [1]. To track the progress, we have created the umbrella issue
> > FLINK-10429
> > [2] (bear in mind that the current sub-tasks are still subject to
> change).
> >
> > We are currently in the process of finalizing the scheduler interfaces.
> Our
> > current state can be found in [3]. Feel free to review and comment on our
> > design proposal.
> >
> > Best,
> > Gary
> >
> > [1]
> >
> >
> https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit
> > [2] https://issues.apache.org/jira/browse/FLINK-10429
> > [3]
> >
> >
> https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU/edit?usp=sharing
> >
>

[Zhu Zhu]

The new interface will be very helpful to extend the scheduling strategy.
It also makes the scheduling and failover process much cleaner.
Thanks Gary to bring up this proposal. I really like it.
+1 for this proposal.

Regards,
Zhu

zhijiang <[hidden email]> 于2019年4月15日周一 下午2:45写道：

> Thanks for sharing the latest updates for the scheduler refactoring Gary.
> It is really good to me and wish seeing the new interface ready soon.
>
> Best,
> Zhijiang
> ------------------------------------------------------------------
> From:Guowei Ma <[hidden email]>
> Send Time:2019年4月15日(星期一) 09:25
> To:dev <[hidden email]>
> Subject:Re: [PROGRESS-UPDATE] Redesign Flink Scheduling, introducing
> dedicated Scheduler Component
>
> Thanks Gary for sharing the documents to the community.
> The idea makes scheduler more flexible.
>
> Best,
> Guowei
>
>
> Till Rohrmann <[hidden email]> 于2019年4月13日周六 上午12:21写道：
>
> > Thanks for sharing the current state of the scheduler refactorings with
> the
> > community Gary. The proposed changes look good to me and, hence, +1 for
> > proceeding with bringing the interfaces into place.
> >
> > Cheers,
> > Till
> >
> > On Fri, Apr 12, 2019 at 5:53 PM Gary Yao <[hidden email]> wrote:
> >
> > > Hi all,
> > >
> > > As you might have seen already, we are currently reworking Flink's
> > > scheduling.
> > > At the moment, scheduling is a concern that is scattered across
> different
> > > components, such as ExecutionGraph, Execution and SlotPool. Scheduling
> > also
> > > happens only on the granularity of individual tasks, which make
> holistic
> > > scheduling strategies hard to implement. For more details on our
> > motivation
> > > see [1]. To track the progress, we have created the umbrella issue
> > > FLINK-10429
> > > [2] (bear in mind that the current sub-tasks are still subject to
> > change).
> > >
> > > We are currently in the process of finalizing the scheduler interfaces.
> > Our
> > > current state can be found in [3]. Feel free to review and comment on
> our
> > > design proposal.
> > >
> > > Best,
> > > Gary
> > >
> > > [1]
> > >
> > >
> >
> https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit
> > > [2] https://issues.apache.org/jira/browse/FLINK-10429
> > > [3]
> > >
> > >
> >
> https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU/edit?usp=sharing
> > >
> >
>
>