[VOTE] FLIP-53: Fine Grained Operator Resource Management

[Xintong Song]

Hi all,

I would like to start the voting process for FLIP-53 [1], which is
discussed and reached consensus in this thread [2].

This voting will be open for at least 72 hours (excluding weekends). I'll
try to close it Sep. 11, 04:00 UTC, unless there is an objection or not
enough votes.

Thank you~

Xintong Song


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management

[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html

[Zhu Zhu]

Thanks Xintong for proposing this better resource management.
This helps a lot to users who want to better manage the job resources. And
would be even more useful if in the future we can have auto-tuning
mechanism for jobs.

+1 (non-binding)

Thanks,
Zhu Zhu

Xintong Song <[hidden email]> 于2019年9月6日周五 上午11:17写道：

> Hi all,
>
> I would like to start the voting process for FLIP-53 [1], which is
> discussed and reached consensus in this thread [2].
>
> This voting will be open for at least 72 hours (excluding weekends). I'll
> try to close it Sep. 11, 04:00 UTC, unless there is an objection or not
> enough votes.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>
> [2]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
>

[Till Rohrmann]

Hi Xintong,

thanks for starting this vote. The proposal looks good and, hence, +1 for
it.

One comment I have is concerning the first implementation step. I would
suggest to not add the flag allSourcesInSamePipelinedRegion to the
ExecutionConfig because the ExecutionConfig is public API. Ideally we keep
this flag internal and don't expose it to the user.

Cheers,
Till

On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu <[hidden email]> wrote:

> Thanks Xintong for proposing this better resource management.
> This helps a lot to users who want to better manage the job resources. And
> would be even more useful if in the future we can have auto-tuning
> mechanism for jobs.
>
> +1 (non-binding)
>
> Thanks,
> Zhu Zhu
>
> Xintong Song <[hidden email]> 于2019年9月6日周五 上午11:17写道：
>
> > Hi all,
> >
> > I would like to start the voting process for FLIP-53 [1], which is
> > discussed and reached consensus in this thread [2].
> >
> > This voting will be open for at least 72 hours (excluding weekends). I'll
> > try to close it Sep. 11, 04:00 UTC, unless there is an objection or not
> > enough votes.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> >
> > [2]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
> >
>

[Andrey Zagrebin-3]

Thanks for starting the vote @Xintong

+1 for the FLIP-53

Best,
Andrey

On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann <[hidden email]> wrote:

> Hi Xintong,
>
> thanks for starting this vote. The proposal looks good and, hence, +1 for
> it.
>
> One comment I have is concerning the first implementation step. I would
> suggest to not add the flag allSourcesInSamePipelinedRegion to the
> ExecutionConfig because the ExecutionConfig is public API. Ideally we keep
> this flag internal and don't expose it to the user.
>
> Cheers,
> Till
>
> On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu <[hidden email]> wrote:
>
> > Thanks Xintong for proposing this better resource management.
> > This helps a lot to users who want to better manage the job resources.
> And
> > would be even more useful if in the future we can have auto-tuning
> > mechanism for jobs.
> >
> > +1 (non-binding)
> >
> > Thanks,
> > Zhu Zhu
> >
> > Xintong Song <[hidden email]> 于2019年9月6日周五 上午11:17写道：
> >
> > > Hi all,
> > >
> > > I would like to start the voting process for FLIP-53 [1], which is
> > > discussed and reached consensus in this thread [2].
> > >
> > > This voting will be open for at least 72 hours (excluding weekends).
> I'll
> > > try to close it Sep. 11, 04:00 UTC, unless there is an objection or not
> > > enough votes.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > >
> > > [2]
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
> > >
> >
>

[Kurt Young]

+1 for FLIP-53.

I would like to raise one minor concern regarding to implementing
request absolute amount of memory case. Currently, it will be
translated to a memory fraction during compile, and translate back
to absolute value during execution. There is a risk that the user might
get less than he requested due to floating point number problems.

Best,
Kurt


On Fri, Sep 6, 2019 at 10:13 PM Andrey Zagrebin <[hidden email]>
wrote:

> Thanks for starting the vote @Xintong
>
> +1 for the FLIP-53
>
> Best,
> Andrey
>
> On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann <[hidden email]> wrote:
>
> > Hi Xintong,
> >
> > thanks for starting this vote. The proposal looks good and, hence, +1 for
> > it.
> >
> > One comment I have is concerning the first implementation step. I would
> > suggest to not add the flag allSourcesInSamePipelinedRegion to the
> > ExecutionConfig because the ExecutionConfig is public API. Ideally we
> keep
> > this flag internal and don't expose it to the user.
> >
> > Cheers,
> > Till
> >
> > On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu <[hidden email]> wrote:
> >
> > > Thanks Xintong for proposing this better resource management.
> > > This helps a lot to users who want to better manage the job resources.
> > And
> > > would be even more useful if in the future we can have auto-tuning
> > > mechanism for jobs.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Xintong Song <[hidden email]> 于2019年9月6日周五 上午11:17写道：
> > >
> > > > Hi all,
> > > >
> > > > I would like to start the voting process for FLIP-53 [1], which is
> > > > discussed and reached consensus in this thread [2].
> > > >
> > > > This voting will be open for at least 72 hours (excluding weekends).
> > I'll
> > > > try to close it Sep. 11, 04:00 UTC, unless there is an objection or
> not
> > > > enough votes.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > > >
> > > > [2]
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
> > > >
> > >
> >
>

[Xintong Song]

+1 from my side.

Thanks for bringing this up, Kurt. It's true that the accuracy loss of
floating values could cause problems in some cases. And I would consider
this as an implementation issue that can be discussed later in the PRs.

Thank you~

Xintong Song



On Sat, Sep 7, 2019 at 2:51 PM Kurt Young <[hidden email]> wrote:

> +1 for FLIP-53.
>
> I would like to raise one minor concern regarding to implementing
> request absolute amount of memory case. Currently, it will be
> translated to a memory fraction during compile, and translate back
> to absolute value during execution. There is a risk that the user might
> get less than he requested due to floating point number problems.
>
> Best,
> Kurt
>
>
> On Fri, Sep 6, 2019 at 10:13 PM Andrey Zagrebin <[hidden email]>
> wrote:
>
> > Thanks for starting the vote @Xintong
> >
> > +1 for the FLIP-53
> >
> > Best,
> > Andrey
> >
> > On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann <[hidden email]>
> wrote:
> >
> > > Hi Xintong,
> > >
> > > thanks for starting this vote. The proposal looks good and, hence, +1
> for
> > > it.
> > >
> > > One comment I have is concerning the first implementation step. I would
> > > suggest to not add the flag allSourcesInSamePipelinedRegion to the
> > > ExecutionConfig because the ExecutionConfig is public API. Ideally we
> > keep
> > > this flag internal and don't expose it to the user.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu <[hidden email]> wrote:
> > >
> > > > Thanks Xintong for proposing this better resource management.
> > > > This helps a lot to users who want to better manage the job
> resources.
> > > And
> > > > would be even more useful if in the future we can have auto-tuning
> > > > mechanism for jobs.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > Xintong Song <[hidden email]> 于2019年9月6日周五 上午11:17写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start the voting process for FLIP-53 [1], which is
> > > > > discussed and reached consensus in this thread [2].
> > > > >
> > > > > This voting will be open for at least 72 hours (excluding
> weekends).
> > > I'll
> > > > > try to close it Sep. 11, 04:00 UTC, unless there is an objection or
> > not
> > > > > enough votes.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
> > > > >
> > > >
> > >
> >
>

[Xintong Song]

Thanks all for the votes.

So far, we have

   - 3 binding +1 votes (Till, Andrey and Kurt)
   - 2 un-binding +1 votes (Zhu and Xintong)
   - No -1 votes

There are 3 binding +1 votes and no -1 votes, and the voting time has past.
According to the community bylaws, I'm glad to announce that FLIP-53 is
approved to be adopted by Apache Flink.

Thank you~

Xintong Song



On Thu, Sep 12, 2019 at 10:31 AM Xintong Song <[hidden email]> wrote:

> +1 from my side.
>
> Thanks for bringing this up, Kurt. It's true that the accuracy loss of
> floating values could cause problems in some cases. And I would consider
> this as an implementation issue that can be discussed later in the PRs.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Sat, Sep 7, 2019 at 2:51 PM Kurt Young <[hidden email]> wrote:
>
>> +1 for FLIP-53.
>>
>> I would like to raise one minor concern regarding to implementing
>> request absolute amount of memory case. Currently, it will be
>> translated to a memory fraction during compile, and translate back
>> to absolute value during execution. There is a risk that the user might
>> get less than he requested due to floating point number problems.
>>
>> Best,
>> Kurt
>>
>>
>> On Fri, Sep 6, 2019 at 10:13 PM Andrey Zagrebin <[hidden email]>
>> wrote:
>>
>> > Thanks for starting the vote @Xintong
>> >
>> > +1 for the FLIP-53
>> >
>> > Best,
>> > Andrey
>> >
>> > On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann <[hidden email]>
>> wrote:
>> >
>> > > Hi Xintong,
>> > >
>> > > thanks for starting this vote. The proposal looks good and, hence, +1
>> for
>> > > it.
>> > >
>> > > One comment I have is concerning the first implementation step. I
>> would
>> > > suggest to not add the flag allSourcesInSamePipelinedRegion to the
>> > > ExecutionConfig because the ExecutionConfig is public API. Ideally we
>> > keep
>> > > this flag internal and don't expose it to the user.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu <[hidden email]> wrote:
>> > >
>> > > > Thanks Xintong for proposing this better resource management.
>> > > > This helps a lot to users who want to better manage the job
>> resources.
>> > > And
>> > > > would be even more useful if in the future we can have auto-tuning
>> > > > mechanism for jobs.
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > Thanks,
>> > > > Zhu Zhu
>> > > >
>> > > > Xintong Song <[hidden email]> 于2019年9月6日周五 上午11:17写道：
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I would like to start the voting process for FLIP-53 [1], which is
>> > > > > discussed and reached consensus in this thread [2].
>> > > > >
>> > > > > This voting will be open for at least 72 hours (excluding
>> weekends).
>> > > I'll
>> > > > > try to close it Sep. 11, 04:00 UTC, unless there is an objection
>> or
>> > not
>> > > > > enough votes.
>> > > > >
>> > > > > Thank you~
>> > > > >
>> > > > > Xintong Song
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>> > > > >
>> > > > > [2]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
>> > > > >
>> > > >
>> > >
>> >
>>
>