[VOTE] FLIP-141: Intra-Slot Managed Memory Sharing

[Xintong Song]

Hi devs,

I'd like to start a voting thread on FLIP-141[1], which proposes how
managed memory should be shared by various use cases within a slot. The
proposal has been discussed in [2].

The vote will be open for at least 72h + weekends. I'll try to close it on
September 8, unless there is an objection or not enough votes.

Thank you~

Xintong Song


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility

[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html

[Till Rohrmann]

Hi Xintong,

thanks for starting the vote.

+1 for the proposal given that we find a proper name for the
different memory consumers (specifically the batch/RocksDB consumer) and
their corresponding weights.

Cheers,
Till

On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <[hidden email]> wrote:

> Hi devs,
>
> I'd like to start a voting thread on FLIP-141[1], which proposes how
> managed memory should be shared by various use cases within a slot. The
> proposal has been discussed in [2].
>
> The vote will be open for at least 72h + weekends. I'll try to close it on
> September 8, unless there is an objection or not enough votes.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>
> [2]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>

[Dian Fu-2]

+1

> 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
>
> Hi Xintong,
>
> thanks for starting the vote.
>
> +1 for the proposal given that we find a proper name for the
> different memory consumers (specifically the batch/RocksDB consumer) and
> their corresponding weights.
>
> Cheers,
> Till
>
> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <[hidden email]> wrote:
>
>> Hi devs,
>>
>> I'd like to start a voting thread on FLIP-141[1], which proposes how
>> managed memory should be shared by various use cases within a slot. The
>> proposal has been discussed in [2].
>>
>> The vote will be open for at least 72h + weekends. I'll try to close it on
>> September 8, unless there is an objection or not enough votes.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>>
>> [2]
>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>>

[Zhu Zhu]

+1

Thanks,
Zhu

Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：

> +1
>
> > 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> >
> > Hi Xintong,
> >
> > thanks for starting the vote.
> >
> > +1 for the proposal given that we find a proper name for the
> > different memory consumers (specifically the batch/RocksDB consumer) and
> > their corresponding weights.
> >
> > Cheers,
> > Till
> >
> > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <[hidden email]>
> wrote:
> >
> >> Hi devs,
> >>
> >> I'd like to start a voting thread on FLIP-141[1], which proposes how
> >> managed memory should be shared by various use cases within a slot. The
> >> proposal has been discussed in [2].
> >>
> >> The vote will be open for at least 72h + weekends. I'll try to close it
> on
> >> September 8, unless there is an objection or not enough votes.
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> >>
> >> [2]
> >>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> >>
>
>

[Yangze Guo]

+1

Best,
Yangze Guo

On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:

>
> +1
>
> Thanks,
> Zhu
>
> Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
>
> > +1
> >
> > > 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> > >
> > > Hi Xintong,
> > >
> > > thanks for starting the vote.
> > >
> > > +1 for the proposal given that we find a proper name for the
> > > different memory consumers (specifically the batch/RocksDB consumer) and
> > > their corresponding weights.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <[hidden email]>
> > wrote:
> > >
> > >> Hi devs,
> > >>
> > >> I'd like to start a voting thread on FLIP-141[1], which proposes how
> > >> managed memory should be shared by various use cases within a slot. The
> > >> proposal has been discussed in [2].
> > >>
> > >> The vote will be open for at least 72h + weekends. I'll try to close it
> > on
> > >> September 8, unless there is an objection or not enough votes.
> > >>
> > >> Thank you~
> > >>
> > >> Xintong Song
> > >>
> > >>
> > >> [1]
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > >>
> > >> [2]
> > >>
> > >>
> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > >>
> >
> >

[Xintong Song]

Thanks for the votes.

Concerning the name for batch/RocksDB memory consumer, how about "execution
memory"?
We can further explain in docs and config option description that this is
used for job execution, which is currently dedicated to rocksdb in
streaming and batch algorithms in batch.

Thank you~

Xintong Song



On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]> wrote:

> +1
>
> Best,
> Yangze Guo
>
> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
> >
> > +1
> >
> > Thanks,
> > Zhu
> >
> > Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
> >
> > > +1
> > >
> > > > 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> > > >
> > > > Hi Xintong,
> > > >
> > > > thanks for starting the vote.
> > > >
> > > > +1 for the proposal given that we find a proper name for the
> > > > different memory consumers (specifically the batch/RocksDB consumer)
> and
> > > > their corresponding weights.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <[hidden email]>
> > > wrote:
> > > >
> > > >> Hi devs,
> > > >>
> > > >> I'd like to start a voting thread on FLIP-141[1], which proposes how
> > > >> managed memory should be shared by various use cases within a slot.
> The
> > > >> proposal has been discussed in [2].
> > > >>
> > > >> The vote will be open for at least 72h + weekends. I'll try to
> close it
> > > on
> > > >> September 8, unless there is an objection or not enough votes.
> > > >>
> > > >> Thank you~
> > > >>
> > > >> Xintong Song
> > > >>
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > > >>
> > > >> [2]
> > > >>
> > > >>
> > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > > >>
> > >
> > >
>

[jincheng sun]

+1 for the proposal!

Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the configuration
names for batch and rocksdb respectively, ` batch_ OP` for batch, "ROCKSDB"
for roockdb. and the default value as follows:

{
    BATCH_OP: 70,
    ROCKSDB : 70,
    PYTHON : 30
}

Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?

Best,
Jincheng


Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：

> Thanks for the votes.
>
> Concerning the name for batch/RocksDB memory consumer, how about "execution
> memory"?
> We can further explain in docs and config option description that this is
> used for job execution, which is currently dedicated to rocksdb in
> streaming and batch algorithms in batch.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]> wrote:
>
> > +1
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
> > >
> > > +1
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
> > >
> > > > +1
> > > >
> > > > > 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> > > > >
> > > > > Hi Xintong,
> > > > >
> > > > > thanks for starting the vote.
> > > > >
> > > > > +1 for the proposal given that we find a proper name for the
> > > > > different memory consumers (specifically the batch/RocksDB
> consumer)
> > and
> > > > > their corresponding weights.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> [hidden email]>
> > > > wrote:
> > > > >
> > > > >> Hi devs,
> > > > >>
> > > > >> I'd like to start a voting thread on FLIP-141[1], which proposes
> how
> > > > >> managed memory should be shared by various use cases within a
> slot.
> > The
> > > > >> proposal has been discussed in [2].
> > > > >>
> > > > >> The vote will be open for at least 72h + weekends. I'll try to
> > close it
> > > > on
> > > > >> September 8, unless there is an objection or not enough votes.
> > > > >>
> > > > >> Thank you~
> > > > >>
> > > > >> Xintong Song
> > > > >>
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > > > >>
> > > > >> [2]
> > > > >>
> > > > >>
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > > > >>
> > > >
> > > >
> >
>

[Xintong Song]

Thanks for the vote, @Jincheng.


Concerning the namings, the original idea was, as you suggested, to have
separate configuration names for batch and rocksdb while only one of them
will take effect at a time.


It was then in the discussion thread [1] that @Stepahn suggested to combine
these two.

>     We never have batch algos and RocksDB mixed, having this as separate
> options is confusing as it suggests this can be combined arbitrarily. I
> also think that a slim possibility that we may ever combine this in the
> future is not enough reason to make it more complex/confusing.
>

This suggestion was also supported by others in the discussion thread.
That's why we are trying to come up with a name that covers both batch and
rocksdb memory consumers.


Thank you~

Xintong Song


[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html

On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <[hidden email]>
wrote:

> +1 for the proposal!
>
> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the configuration
> names for batch and rocksdb respectively, ` batch_ OP` for batch, "ROCKSDB"
> for roockdb. and the default value as follows:
>
> {
>     BATCH_OP: 70,
>     ROCKSDB : 70,
>     PYTHON : 30
> }
>
> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>
> Best,
> Jincheng
>
>
> Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：
>
> > Thanks for the votes.
> >
> > Concerning the name for batch/RocksDB memory consumer, how about
> "execution
> > memory"?
> > We can further explain in docs and config option description that this is
> > used for job execution, which is currently dedicated to rocksdb in
> > streaming and batch algorithms in batch.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]> wrote:
> >
> > > +1
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
> > > >
> > > > +1
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
> > > >
> > > > > +1
> > > > >
> > > > > > 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> > > > > >
> > > > > > Hi Xintong,
> > > > > >
> > > > > > thanks for starting the vote.
> > > > > >
> > > > > > +1 for the proposal given that we find a proper name for the
> > > > > > different memory consumers (specifically the batch/RocksDB
> > consumer)
> > > and
> > > > > > their corresponding weights.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> > [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > >> Hi devs,
> > > > > >>
> > > > > >> I'd like to start a voting thread on FLIP-141[1], which proposes
> > how
> > > > > >> managed memory should be shared by various use cases within a
> > slot.
> > > The
> > > > > >> proposal has been discussed in [2].
> > > > > >>
> > > > > >> The vote will be open for at least 72h + weekends. I'll try to
> > > close it
> > > > > on
> > > > > >> September 8, unless there is an objection or not enough votes.
> > > > > >>
> > > > > >> Thank you~
> > > > > >>
> > > > > >> Xintong Song
> > > > > >>
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > > > > >>
> > > > > >> [2]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > > > > >>
> > > > >
> > > > >
> > >
> >
>

[Aljoscha Krettek-2]

+1

We just need to make sure to find a good name before the release but
shouldn't block any work on this.

Aljoscha

On 08.09.20 07:59, Xintong Song wrote:

> Thanks for the vote, @Jincheng.
>
>
> Concerning the namings, the original idea was, as you suggested, to have
> separate configuration names for batch and rocksdb while only one of them
> will take effect at a time.
>
>
> It was then in the discussion thread [1] that @Stepahn suggested to combine
> these two.
>
>>      We never have batch algos and RocksDB mixed, having this as separate
>> options is confusing as it suggests this can be combined arbitrarily. I
>> also think that a slim possibility that we may ever combine this in the
>> future is not enough reason to make it more complex/confusing.
>>
>
> This suggestion was also supported by others in the discussion thread.
> That's why we are trying to come up with a name that covers both batch and
> rocksdb memory consumers.
>
>
> Thank you~
>
> Xintong Song
>
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
>
> On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <[hidden email]>
> wrote:
>
>> +1 for the proposal!
>>
>> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the configuration
>> names for batch and rocksdb respectively, ` batch_ OP` for batch, "ROCKSDB"
>> for roockdb. and the default value as follows:
>>
>> {
>>      BATCH_OP: 70,
>>      ROCKSDB : 70,
>>      PYTHON : 30
>> }
>>
>> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>>
>> Best,
>> Jincheng
>>
>>
>> Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：
>>
>>> Thanks for the votes.
>>>
>>> Concerning the name for batch/RocksDB memory consumer, how about
>> "execution
>>> memory"?
>>> We can further explain in docs and config option description that this is
>>> used for job execution, which is currently dedicated to rocksdb in
>>> streaming and batch algorithms in batch.
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]> wrote:
>>>
>>>> +1
>>>>
>>>> Best,
>>>> Yangze Guo
>>>>
>>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>> Thanks,
>>>>> Zhu
>>>>>
>>>>> Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
>>>>>
>>>>>> +1
>>>>>>
>>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
>>>>>>>
>>>>>>> Hi Xintong,
>>>>>>>
>>>>>>> thanks for starting the vote.
>>>>>>>
>>>>>>> +1 for the proposal given that we find a proper name for the
>>>>>>> different memory consumers (specifically the batch/RocksDB
>>> consumer)
>>>> and
>>>>>>> their corresponding weights.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
>>> [hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi devs,
>>>>>>>>
>>>>>>>> I'd like to start a voting thread on FLIP-141[1], which proposes
>>> how
>>>>>>>> managed memory should be shared by various use cases within a
>>> slot.
>>>> The
>>>>>>>> proposal has been discussed in [2].
>>>>>>>>
>>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
>>>> close it
>>>>>> on
>>>>>>>> September 8, unless there is an objection or not enough votes.
>>>>>>>>
>>>>>>>> Thank you~
>>>>>>>>
>>>>>>>> Xintong Song
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>>>>>>>>
>>>>>>>> [2]
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>

[Yu Li]

+1

Best Regards,
Yu


On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <[hidden email]> wrote:

> +1
>
> We just need to make sure to find a good name before the release but
> shouldn't block any work on this.
>
> Aljoscha
>
> On 08.09.20 07:59, Xintong Song wrote:
> > Thanks for the vote, @Jincheng.
> >
> >
> > Concerning the namings, the original idea was, as you suggested, to have
> > separate configuration names for batch and rocksdb while only one of them
> > will take effect at a time.
> >
> >
> > It was then in the discussion thread [1] that @Stepahn suggested to
> combine
> > these two.
> >
> >>      We never have batch algos and RocksDB mixed, having this as
> separate
> >> options is confusing as it suggests this can be combined arbitrarily. I
> >> also think that a slim possibility that we may ever combine this in the
> >> future is not enough reason to make it more complex/confusing.
> >>
> >
> > This suggestion was also supported by others in the discussion thread.
> > That's why we are trying to come up with a name that covers both batch
> and
> > rocksdb memory consumers.
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> >
> > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <[hidden email]>
> > wrote:
> >
> >> +1 for the proposal!
> >>
> >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> configuration
> >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> "ROCKSDB"
> >> for roockdb. and the default value as follows:
> >>
> >> {
> >>      BATCH_OP: 70,
> >>      ROCKSDB : 70,
> >>      PYTHON : 30
> >> }
> >>
> >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> >>
> >> Best,
> >> Jincheng
> >>
> >>
> >> Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：
> >>
> >>> Thanks for the votes.
> >>>
> >>> Concerning the name for batch/RocksDB memory consumer, how about
> >> "execution
> >>> memory"?
> >>> We can further explain in docs and config option description that this
> is
> >>> used for job execution, which is currently dedicated to rocksdb in
> >>> streaming and batch algorithms in batch.
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
> >>>
> >>>
> >>>
> >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> Best,
> >>>> Yangze Guo
> >>>>
> >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
> >>>>>
> >>>>> +1
> >>>>>
> >>>>> Thanks,
> >>>>> Zhu
> >>>>>
> >>>>> Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> >>>>>>>
> >>>>>>> Hi Xintong,
> >>>>>>>
> >>>>>>> thanks for starting the vote.
> >>>>>>>
> >>>>>>> +1 for the proposal given that we find a proper name for the
> >>>>>>> different memory consumers (specifically the batch/RocksDB
> >>> consumer)
> >>>> and
> >>>>>>> their corresponding weights.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>>
> >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> >>> [hidden email]>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi devs,
> >>>>>>>>
> >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which proposes
> >>> how
> >>>>>>>> managed memory should be shared by various use cases within a
> >>> slot.
> >>>> The
> >>>>>>>> proposal has been discussed in [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
> >>>> close it
> >>>>>> on
> >>>>>>>> September 8, unless there is an objection or not enough votes.
> >>>>>>>>
> >>>>>>>> Thank you~
> >>>>>>>>
> >>>>>>>> Xintong Song
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> >>>>>>>>
> >>>>>>>> [2]
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> >
>
>

[Andrey Zagrebin-4]

+1

Best,
Andrey

On Tue, Sep 8, 2020 at 2:16 PM Yu Li <[hidden email]> wrote:

> +1
>
> Best Regards,
> Yu
>
>
> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <[hidden email]> wrote:
>
> > +1
> >
> > We just need to make sure to find a good name before the release but
> > shouldn't block any work on this.
> >
> > Aljoscha
> >
> > On 08.09.20 07:59, Xintong Song wrote:
> > > Thanks for the vote, @Jincheng.
> > >
> > >
> > > Concerning the namings, the original idea was, as you suggested, to
> have
> > > separate configuration names for batch and rocksdb while only one of
> them
> > > will take effect at a time.
> > >
> > >
> > > It was then in the discussion thread [1] that @Stepahn suggested to
> > combine
> > > these two.
> > >
> > >>      We never have batch algos and RocksDB mixed, having this as
> > separate
> > >> options is confusing as it suggests this can be combined arbitrarily.
> I
> > >> also think that a slim possibility that we may ever combine this in
> the
> > >> future is not enough reason to make it more complex/confusing.
> > >>
> > >
> > > This suggestion was also supported by others in the discussion thread.
> > > That's why we are trying to come up with a name that covers both batch
> > and
> > > rocksdb memory consumers.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> > >
> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <[hidden email]>
> > > wrote:
> > >
> > >> +1 for the proposal!
> > >>
> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> > configuration
> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> > "ROCKSDB"
> > >> for roockdb. and the default value as follows:
> > >>
> > >> {
> > >>      BATCH_OP: 70,
> > >>      ROCKSDB : 70,
> > >>      PYTHON : 30
> > >> }
> > >>
> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> > >>
> > >> Best,
> > >> Jincheng
> > >>
> > >>
> > >> Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：
> > >>
> > >>> Thanks for the votes.
> > >>>
> > >>> Concerning the name for batch/RocksDB memory consumer, how about
> > >> "execution
> > >>> memory"?
> > >>> We can further explain in docs and config option description that
> this
> > is
> > >>> used for job execution, which is currently dedicated to rocksdb in
> > >>> streaming and batch algorithms in batch.
> > >>>
> > >>> Thank you~
> > >>>
> > >>> Xintong Song
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]>
> wrote:
> > >>>
> > >>>> +1
> > >>>>
> > >>>> Best,
> > >>>> Yangze Guo
> > >>>>
> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
> > >>>>>
> > >>>>> +1
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Zhu
> > >>>>>
> > >>>>> Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
> > >>>>>
> > >>>>>> +1
> > >>>>>>
> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> > >>>>>>>
> > >>>>>>> Hi Xintong,
> > >>>>>>>
> > >>>>>>> thanks for starting the vote.
> > >>>>>>>
> > >>>>>>> +1 for the proposal given that we find a proper name for the
> > >>>>>>> different memory consumers (specifically the batch/RocksDB
> > >>> consumer)
> > >>>> and
> > >>>>>>> their corresponding weights.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> > >>> [hidden email]>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi devs,
> > >>>>>>>>
> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which proposes
> > >>> how
> > >>>>>>>> managed memory should be shared by various use cases within a
> > >>> slot.
> > >>>> The
> > >>>>>>>> proposal has been discussed in [2].
> > >>>>>>>>
> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
> > >>>> close it
> > >>>>>> on
> > >>>>>>>> September 8, unless there is an objection or not enough votes.
> > >>>>>>>>
> > >>>>>>>> Thank you~
> > >>>>>>>>
> > >>>>>>>> Xintong Song
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > >>>>>>>>
> > >>>>>>>> [2]
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>

[Andrey Zagrebin-4]

For the option name, maybe:
*flink.main*
or
*flink.managed* (this may be a bit confusing for existing users as we said
that the overall managed memory is managed by Flink)

On Wed, Sep 9, 2020 at 9:56 AM Andrey Zagrebin <[hidden email]> wrote:

> +1
>
> Best,
> Andrey
>
> On Tue, Sep 8, 2020 at 2:16 PM Yu Li <[hidden email]> wrote:
>
>> +1
>>
>> Best Regards,
>> Yu
>>
>>
>> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <[hidden email]>
>> wrote:
>>
>> > +1
>> >
>> > We just need to make sure to find a good name before the release but
>> > shouldn't block any work on this.
>> >
>> > Aljoscha
>> >
>> > On 08.09.20 07:59, Xintong Song wrote:
>> > > Thanks for the vote, @Jincheng.
>> > >
>> > >
>> > > Concerning the namings, the original idea was, as you suggested, to
>> have
>> > > separate configuration names for batch and rocksdb while only one of
>> them
>> > > will take effect at a time.
>> > >
>> > >
>> > > It was then in the discussion thread [1] that @Stepahn suggested to
>> > combine
>> > > these two.
>> > >
>> > >>      We never have batch algos and RocksDB mixed, having this as
>> > separate
>> > >> options is confusing as it suggests this can be combined
>> arbitrarily. I
>> > >> also think that a slim possibility that we may ever combine this in
>> the
>> > >> future is not enough reason to make it more complex/confusing.
>> > >>
>> > >
>> > > This suggestion was also supported by others in the discussion thread.
>> > > That's why we are trying to come up with a name that covers both batch
>> > and
>> > > rocksdb memory consumers.
>> > >
>> > >
>> > > Thank you~
>> > >
>> > > Xintong Song
>> > >
>> > >
>> > > [1]
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
>> > >
>> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <[hidden email]
>> >
>> > > wrote:
>> > >
>> > >> +1 for the proposal!
>> > >>
>> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
>> > configuration
>> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
>> > "ROCKSDB"
>> > >> for roockdb. and the default value as follows:
>> > >>
>> > >> {
>> > >>      BATCH_OP: 70,
>> > >>      ROCKSDB : 70,
>> > >>      PYTHON : 30
>> > >> }
>> > >>
>> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>> > >>
>> > >> Best,
>> > >> Jincheng
>> > >>
>> > >>
>> > >> Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：
>> > >>
>> > >>> Thanks for the votes.
>> > >>>
>> > >>> Concerning the name for batch/RocksDB memory consumer, how about
>> > >> "execution
>> > >>> memory"?
>> > >>> We can further explain in docs and config option description that
>> this
>> > is
>> > >>> used for job execution, which is currently dedicated to rocksdb in
>> > >>> streaming and batch algorithms in batch.
>> > >>>
>> > >>> Thank you~
>> > >>>
>> > >>> Xintong Song
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]>
>> wrote:
>> > >>>
>> > >>>> +1
>> > >>>>
>> > >>>> Best,
>> > >>>> Yangze Guo
>> > >>>>
>> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]> wrote:
>> > >>>>>
>> > >>>>> +1
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Zhu
>> > >>>>>
>> > >>>>> Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
>> > >>>>>
>> > >>>>>> +1
>> > >>>>>>
>> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
>> > >>>>>>>
>> > >>>>>>> Hi Xintong,
>> > >>>>>>>
>> > >>>>>>> thanks for starting the vote.
>> > >>>>>>>
>> > >>>>>>> +1 for the proposal given that we find a proper name for the
>> > >>>>>>> different memory consumers (specifically the batch/RocksDB
>> > >>> consumer)
>> > >>>> and
>> > >>>>>>> their corresponding weights.
>> > >>>>>>>
>> > >>>>>>> Cheers,
>> > >>>>>>> Till
>> > >>>>>>>
>> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
>> > >>> [hidden email]>
>> > >>>>>> wrote:
>> > >>>>>>>
>> > >>>>>>>> Hi devs,
>> > >>>>>>>>
>> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which
>> proposes
>> > >>> how
>> > >>>>>>>> managed memory should be shared by various use cases within a
>> > >>> slot.
>> > >>>> The
>> > >>>>>>>> proposal has been discussed in [2].
>> > >>>>>>>>
>> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
>> > >>>> close it
>> > >>>>>> on
>> > >>>>>>>> September 8, unless there is an objection or not enough votes.
>> > >>>>>>>>
>> > >>>>>>>> Thank you~
>> > >>>>>>>>
>> > >>>>>>>> Xintong Song
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> [1]
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>> > >>>>>>>>
>> > >>>>>>>> [2]
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> >
>> >
>>
>

[Xintong Song]

Thanks everyone,

I'm closing this vote now in a separate email.

Concerning the naming, I will use DATAPROC, as @Stephan suggested in the
discussion thread [1], for now. If there are any other opinions, feel free
to reach out to me anytime before the release.

Thank you~

Xintong Song


[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44533.html

On Wed, Sep 9, 2020 at 4:02 PM Andrey Zagrebin <[hidden email]> wrote:

> For the option name, maybe:
> *flink.main*
> or
> *flink.managed* (this may be a bit confusing for existing users as we said
> that the overall managed memory is managed by Flink)
>
> On Wed, Sep 9, 2020 at 9:56 AM Andrey Zagrebin <[hidden email]>
> wrote:
>
> > +1
> >
> > Best,
> > Andrey
> >
> > On Tue, Sep 8, 2020 at 2:16 PM Yu Li <[hidden email]> wrote:
> >
> >> +1
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek <[hidden email]>
> >> wrote:
> >>
> >> > +1
> >> >
> >> > We just need to make sure to find a good name before the release but
> >> > shouldn't block any work on this.
> >> >
> >> > Aljoscha
> >> >
> >> > On 08.09.20 07:59, Xintong Song wrote:
> >> > > Thanks for the vote, @Jincheng.
> >> > >
> >> > >
> >> > > Concerning the namings, the original idea was, as you suggested, to
> >> have
> >> > > separate configuration names for batch and rocksdb while only one of
> >> them
> >> > > will take effect at a time.
> >> > >
> >> > >
> >> > > It was then in the discussion thread [1] that @Stepahn suggested to
> >> > combine
> >> > > these two.
> >> > >
> >> > >>      We never have batch algos and RocksDB mixed, having this as
> >> > separate
> >> > >> options is confusing as it suggests this can be combined
> >> arbitrarily. I
> >> > >> also think that a slim possibility that we may ever combine this in
> >> the
> >> > >> future is not enough reason to make it more complex/confusing.
> >> > >>
> >> > >
> >> > > This suggestion was also supported by others in the discussion
> thread.
> >> > > That's why we are trying to come up with a name that covers both
> batch
> >> > and
> >> > > rocksdb memory consumers.
> >> > >
> >> > >
> >> > > Thank you~
> >> > >
> >> > > Xintong Song
> >> > >
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> >> > >
> >> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun <
> [hidden email]
> >> >
> >> > > wrote:
> >> > >
> >> > >> +1 for the proposal!
> >> > >>
> >> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> >> > configuration
> >> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> >> > "ROCKSDB"
> >> > >> for roockdb. and the default value as follows:
> >> > >>
> >> > >> {
> >> > >>      BATCH_OP: 70,
> >> > >>      ROCKSDB : 70,
> >> > >>      PYTHON : 30
> >> > >> }
> >> > >>
> >> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> >> > >>
> >> > >> Best,
> >> > >> Jincheng
> >> > >>
> >> > >>
> >> > >> Xintong Song <[hidden email]> 于2020年9月7日周一 下午1:46写道：
> >> > >>
> >> > >>> Thanks for the votes.
> >> > >>>
> >> > >>> Concerning the name for batch/RocksDB memory consumer, how about
> >> > >> "execution
> >> > >>> memory"?
> >> > >>> We can further explain in docs and config option description that
> >> this
> >> > is
> >> > >>> used for job execution, which is currently dedicated to rocksdb in
> >> > >>> streaming and batch algorithms in batch.
> >> > >>>
> >> > >>> Thank you~
> >> > >>>
> >> > >>> Xintong Song
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo <[hidden email]>
> >> wrote:
> >> > >>>
> >> > >>>> +1
> >> > >>>>
> >> > >>>> Best,
> >> > >>>> Yangze Guo
> >> > >>>>
> >> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu <[hidden email]>
> wrote:
> >> > >>>>>
> >> > >>>>> +1
> >> > >>>>>
> >> > >>>>> Thanks,
> >> > >>>>> Zhu
> >> > >>>>>
> >> > >>>>> Dian Fu <[hidden email]> 于2020年9月7日周一 上午10:34写道：
> >> > >>>>>
> >> > >>>>>> +1
> >> > >>>>>>
> >> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann <[hidden email]> 写道：
> >> > >>>>>>>
> >> > >>>>>>> Hi Xintong,
> >> > >>>>>>>
> >> > >>>>>>> thanks for starting the vote.
> >> > >>>>>>>
> >> > >>>>>>> +1 for the proposal given that we find a proper name for the
> >> > >>>>>>> different memory consumers (specifically the batch/RocksDB
> >> > >>> consumer)
> >> > >>>> and
> >> > >>>>>>> their corresponding weights.
> >> > >>>>>>>
> >> > >>>>>>> Cheers,
> >> > >>>>>>> Till
> >> > >>>>>>>
> >> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> >> > >>> [hidden email]>
> >> > >>>>>> wrote:
> >> > >>>>>>>
> >> > >>>>>>>> Hi devs,
> >> > >>>>>>>>
> >> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which
> >> proposes
> >> > >>> how
> >> > >>>>>>>> managed memory should be shared by various use cases within a
> >> > >>> slot.
> >> > >>>> The
> >> > >>>>>>>> proposal has been discussed in [2].
> >> > >>>>>>>>
> >> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try
> to
> >> > >>>> close it
> >> > >>>>>> on
> >> > >>>>>>>> September 8, unless there is an objection or not enough
> votes.
> >> > >>>>>>>>
> >> > >>>>>>>> Thank you~
> >> > >>>>>>>>
> >> > >>>>>>>> Xintong Song
> >> > >>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>> [1]
> >> > >>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> >> > >>>>>>>>
> >> > >>>>>>>> [2]
> >> > >>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> >> > >>>>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> > >
> >> >
> >> >
> >>
> >
>