(DEPRECATED) Apache Flink Mailing List archive.

[VOTE] FLIP-101: Add Pending Slots Detail

Classic

List

Threaded

7 messages Options

Yadong Xie

[VOTE] FLIP-101: Add Pending Slots Detail

Hi all

I want to start the vote for FLIP-101, which proposes to add pending slots
information to help users check which vertex/subtask is blocked.

To help everyone better understand the proposal, we spent some efforts on
making an online POC

previous web:
http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
POC web:
http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots

The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

Jark Wu-2

Re: [VOTE] FLIP-101: Add Pending Slots Detail

Thanks Yadong,

I think a pending slot view will be helpful. But will it be verbose when
there is no pending slot, but a "pending slot" in the tab?
What do you think to show the pending slot page when click the "?" on the
vertex status?

Best,
Jark

On Thu, 20 Feb 2020 at 17:50, Yadong Xie <[hidden email]> wrote:

> Hi all
>
> I want to start the vote for FLIP-101, which proposes to add pending slots
> information to help users check which vertex/subtask is blocked.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Kurt Young

Re: [VOTE] FLIP-101: Add Pending Slots Detail

I agree with Jark, even if we have pending slots now, a dedicated tab seems
to be too much.

Best,
Kurt

On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <[hidden email]> wrote:

> Thanks Yadong,
>
> I think a pending slot view will be helpful. But will it be verbose when
> there is no pending slot, but a "pending slot" in the tab?
> What do you think to show the pending slot page when click the "?" on the
> vertex status?
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 17:50, Yadong Xie <[hidden email]> wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-101, which proposes to add pending
> slots
> > information to help users check which vertex/subtask is blocked.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Zhijiang(wangzhijiang999)

Re: [VOTE] FLIP-101: Add Pending Slots Detail

Thanks Yadong for this FLIP!

If I understood correctly, the motivation is based on the scenario of scheduled job state for long time in the well resourced cluster.

It is meaningful for me if we can provide some useful infos to help users analyze the reason. I see from PoC that we can get the
respective vertex which slot is pending and the detailed slot id info.
I am curious that is it feasible to do some valid analysis from logs based on the slot id? Or what's the expectation to use these infos?

Best,
Zhijiang
------------------------------------------------------------------
From:Kurt Young <[hidden email]>
Send Time:2020 Feb. 21 (Fri.) 14:38
To:dev <[hidden email]>
Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail

I agree with Jark, even if we have pending slots now, a dedicated tab seems
to be too much.

Best,
Kurt

On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <[hidden email]> wrote:

jing

Re: [VOTE] FLIP-101: Add Pending Slots Detail

Currently, users can't define slot resources. If there are pending slots,
the main reason is that there are not enough slots, so the slot size is
mainly displayed without more details. If slot resources can be set
dynamically later, more information could be shown.

Zhijiang <[hidden email]> 于2020年2月21日周五下午11:22写道：

> Thanks Yadong for this FLIP!
>
> If I understood correctly, the motivation is based on the scenario of
> scheduled job state for long time in the well resourced cluster.
>
> It is meaningful for me if we can provide some useful infos to help users
> analyze the reason. I see from PoC that we can get the
> respective vertex which slot is pending and the detailed slot id info.
> I am curious that is it feasible to do some valid analysis from logs based
> on the slot id? Or what's the expectation to use these infos?
>
> Best,
> Zhijiang
> ------------------------------------------------------------------
> From:Kurt Young <[hidden email]>
> Send Time:2020 Feb. 21 (Fri.) 14:38
> To:dev <[hidden email]>
> Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
>
> I agree with Jark, even if we have pending slots now, a dedicated tab seems
> to be too much.
>
> Best,
> Kurt
>
>
> On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <[hidden email]> wrote:
>
> > Thanks Yadong,
> >
> > I think a pending slot view will be helpful. But will it be verbose when
> > there is no pending slot, but a "pending slot" in the tab?
> > What do you think to show the pending slot page when click the "?" on the
> > vertex status?
> >
> > Best,
> > Jark
> >
> > On Thu, 20 Feb 2020 at 17:50, Yadong Xie <[hidden email]> wrote:
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-101, which proposes to add pending
> > slots
> > > information to help users check which vertex/subtask is blocked.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>
>

Yadong Xie

Re: [VOTE] FLIP-101: Add Pending Slots Detail

Hi all

Thanks for all your suggestions!

> Hi Jark and Kurt

I have redesigned the UI and move the pending slots information from the
tab to the vertex graph

> Hi Zhijiang

The slots id could not be used to associated with slot detail in the
current version as lining said, we can add more detailed data(popup a modal
or a drawer) in later versions.
I have removed the slots id from the UI to avoid confusing users.

here is the new POC URL:
http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview

any comments are welcome

lining jing <[hidden email]> 于2020年2月24日周一下午4:57写道：

> Currently, users can't define slot resources. If there are pending slots,
> the main reason is that there are not enough slots, so the slot size is
> mainly displayed without more details. If slot resources can be set
> dynamically later, more information could be shown.
>
> Zhijiang <[hidden email]> 于2020年2月21日周五下午11:22写道：
>
> > Thanks Yadong for this FLIP!
> >
> > If I understood correctly, the motivation is based on the scenario of
> > scheduled job state for long time in the well resourced cluster.
> >
> > It is meaningful for me if we can provide some useful infos to help users
> > analyze the reason. I see from PoC that we can get the
> > respective vertex which slot is pending and the detailed slot id info.
> > I am curious that is it feasible to do some valid analysis from logs
> based
> > on the slot id? Or what's the expectation to use these infos?
> >
> > Best,
> > Zhijiang
> > ------------------------------------------------------------------
> > From:Kurt Young <[hidden email]>
> > Send Time:2020 Feb. 21 (Fri.) 14:38
> > To:dev <[hidden email]>
> > Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
> >
> > I agree with Jark, even if we have pending slots now, a dedicated tab
> seems
> > to be too much.
> >
> > Best,
> > Kurt
> >
> >
> > On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <[hidden email]> wrote:
> >
> > > Thanks Yadong,
> > >
> > > I think a pending slot view will be helpful. But will it be verbose
> when
> > > there is no pending slot, but a "pending slot" in the tab?
> > > What do you think to show the pending slot page when click the "?" on
> the
> > > vertex status?
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 20 Feb 2020 at 17:50, Yadong Xie <[hidden email]> wrote:
> > >
> > > > Hi all
> > > >
> > > > I want to start the vote for FLIP-101, which proposes to add pending
> > > slots
> > > > information to help users check which vertex/subtask is blocked.
> > > >
> > > > To help everyone better understand the proposal, we spent some
> efforts
> > on
> > > > making an online POC
> > > >
> > > > previous web:
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > > > POC web:
> > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> > > >
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting
> > > > process.
> > > >
> > > > FLIP wiki:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > >
> > > > Thanks,
> > > >
> > > > Yadong
> > > >
> > >
> >
> >
>

Till Rohrmann

Re: [VOTE] FLIP-101: Add Pending Slots Detail

Hi Yadong,

thanks for creating this FLIP. I like the idea to show the user if some
slots are missing in order to run an operator.

However, since we plan to rework the scheduling part and also how resources
are acquired extensively, I'm not entirely sure whether we should add this
functionality now. I think with the introduction of pipelined region
scheduling and declarative resource management, many implementation
details, e.g. pending slot requests, the decision with which parallelism
the operator should run, etc., will change. Hence, I would suggest to
postpone this change until we have implemented the scheduler changes.

A minor comment concerning API design: PendingSlotRequest is an internal
class which we should not expose via an RPC call. If this is really
necessary, then we should introduce a new type carrying only the required
information (excluding the assigned future, for example).

Cheers,
Till

On Tue, Feb 25, 2020 at 8:29 AM Yadong Xie <[hidden email]> wrote:

> Hi all
>
> Thanks for all your suggestions!
>
> > Hi Jark and Kurt
>
> I have redesigned the UI and move the pending slots information from the
> tab to the vertex graph
>
> > Hi Zhijiang
>
> The slots id could not be used to associated with slot detail in the
> current version as lining said, we can add more detailed data(popup a modal
> or a drawer) in later versions.
> I have removed the slots id from the UI to avoid confusing users.
>
>
> here is the new POC URL:
>
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
>
> any comments are welcome
>
> lining jing <[hidden email]> 于2020年2月24日周一下午4:57写道：
>
> > Currently, users can't define slot resources. If there are pending slots,
> > the main reason is that there are not enough slots, so the slot size is
> > mainly displayed without more details. If slot resources can be set
> > dynamically later, more information could be shown.
> >
> > Zhijiang <[hidden email]> 于2020年2月21日周五下午11:22写道：
> >
> > > Thanks Yadong for this FLIP!
> > >
> > > If I understood correctly, the motivation is based on the scenario of
> > > scheduled job state for long time in the well resourced cluster.
> > >
> > > It is meaningful for me if we can provide some useful infos to help
> users
> > > analyze the reason. I see from PoC that we can get the
> > > respective vertex which slot is pending and the detailed slot id info.
> > > I am curious that is it feasible to do some valid analysis from logs
> > based
> > > on the slot id? Or what's the expectation to use these infos?
> > >
> > > Best,
> > > Zhijiang
> > > ------------------------------------------------------------------
> > > From:Kurt Young <[hidden email]>
> > > Send Time:2020 Feb. 21 (Fri.) 14:38
> > > To:dev <[hidden email]>
> > > Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
> > >
> > > I agree with Jark, even if we have pending slots now, a dedicated tab
> > seems
> > > to be too much.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <[hidden email]> wrote:
> > >
> > > > Thanks Yadong,
> > > >
> > > > I think a pending slot view will be helpful. But will it be verbose
> > when
> > > > there is no pending slot, but a "pending slot" in the tab?
> > > > What do you think to show the pending slot page when click the "?" on
> > the
> > > > vertex status?
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Thu, 20 Feb 2020 at 17:50, Yadong Xie <[hidden email]>
> wrote:
> > > >
> > > > > Hi all
> > > > >
> > > > > I want to start the vote for FLIP-101, which proposes to add
> pending
> > > > slots
> > > > > information to help users check which vertex/subtask is blocked.
> > > > >
> > > > > To help everyone better understand the proposal, we spent some
> > efforts
> > > on
> > > > > making an online POC
> > > > >
> > > > > previous web:
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > > > > POC web:
> > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> > > > >
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > > > process.
> > > > >
> > > > > FLIP wiki:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yadong
> > > > >
> > > >
> > >
> > >
> >
>