[DISCUSS] Releasing Flink 1.8.3

[Hequn Cheng]

That's great, thank you very much! Ideally, we can kick off the release
vote for the first RC of 1.8.3 within next week. :)

On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <[hidden email]> wrote:

> I'm not aware of any more planned changes to flink-shaded; so we could
> start the release right away.
>
> On 15/11/2019 13:44, Hequn Cheng wrote:
> > Hi,
> >
> > @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach for
> > 1.8/1.9.
> > @Ufuk Thank you for the nice summary.
> >
> > Looks good so far except that we need to postpone 1.8.3 a bit to first
> do a
> > flink-shaded release.
> > BTW, @chesnay when would we plan to release the flink-shaded with
> upgraded
> > Jackson?
> >
> > Best, Hequn
> >
> > On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <[hidden email]>
> wrote:
> >
> >> One small modification: the flink-shaded upgrade does not have to be
> >> part of the profile; since it is only intended for internal use anyway
> >> (and thus has limited exposure) we can be pretty sure this doesn't break
> >> anything.
> >>
> >> On 15/11/2019 12:23, Chesnay Schepler wrote:
> >>> Ufuk's summary is correct.
> >>>
> >>> There's a slight caveat in that we'd also have to bump the
> >>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
> >>> but I have no concerns about this change.
> >>>
> >>> On 15/11/2019 12:19, Ufuk Celebi wrote:
> >>>> The opt-in approach seems reasonable to me. +1 to include the
> >>>> profiles in
> >>>> 1.8 and 1.9 without changing the default versions (including the
> default
> >>>> version of flink-shaded).
> >>>>
> >>>> As far as I can tell, the next steps would be:
> >>>>
> >>>> 1) Release flink-shaded with upgraded Jackson
> >>>> 2a) Bump the flink-shaded version by default in master
> >>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles
> >>>> should also
> >>>> cover the upgrade to the most recent flink-shaded version)
> >>>>
> >>>> @Chesnay: is this a correct summary?
> >>>>
> >>>> Note this would block the 1.8.3 release on step 1. As an upside, we
> >>>> might
> >>>> get some additional feedback until the 1.10 release with these
> >>>> profiles in
> >>>> case users make use of them with 1.8/1.9.
> >>>>
> >>>> – Ufuk
> >>>>
> >>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <[hidden email]
> >
> >>>> wrote:
> >>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on master
> >>>>> (and
> >>>>> thus starting from 1.10.0) it's not opt-in.
> >>>>>
> >>>>> I have only proposed it as an opt-in because a) we usually do not
> bump
> >>>>> dependencies in bugfix releases and b) it's a short-term change that
> we
> >>>>> aren't allowing to mature properly.
> >>>>> In contrast, the 1.10 release is significantly further away, hence no
> >>>>> opt-in.
> >>>>>
> >>>>> Hence, I'm not concerned about such kind of ugprades being more
> common
> >>>>> in the future.
> >>>>>
> >>>>> We can certainly support every jackson version that fixes these
> >>>>> vulnerabilities; individual modules can always use a different
> version
> >>>>> (that hopefully includes the fixes).
> >>>>> Ideally of course we'd only be using 1 version, but that may or may
> not
> >>>>> be feasible.
> >>>>>
> >>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
> >>>>>> Hi Chesnay,
> >>>>>>
> >>>>>> Great to hear that jackson-2.10.1 works well on master. Really a
> good
> >>>> job!
> >>>>>> - Whether backport this change to 1.8/1.9
> >>>>>> I had taken a quick look at the security vulnerabilities, some of
> them
> >>>>>> seem can lead to high-security problems, thus from my point of view,
> >>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would like
> to
> >>>>>> trust your judgment as you are more professional at this problem.
> >>>>>>
> >>>>>> - How to port this change to 1.8/1.9
> >>>>>> I think providing an opt-in upgrade is a good idea. Another question
> >>>>>> here is whether do we plan to support multi jackson versions that
> have
> >>>>>> eliminated the security vulnerabilities. If we only plan to support
> >>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an option,
> >>>>>> users can downgrade the flink version if meet problems using the new
> >>>>>> version. Of course, we will try our best to make the new release out
> >>>>>> of question.
> >>>>>> Another concern of making it an opt-in upgrade is, it will make our
> >>>>>> build unlikely convergence as more and more build options will be
> >>>>>> added when we upgrade a commonly used lib like this one.
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>> Best, Hequn
> >>>>>>
> >>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
> [hidden email]
> >>>>>> <mailto:[hidden email]>> wrote:
> >>>>>>
> >>>>>>       So here's the state of things:
> >>>>>>
> >>>>>>
> >>>>>>       The master of flink-shaded now uses jackson 2.10.1, which
> >>>>>>       eliminates a whole category of security vulnerabilities.
> >>>>>>       The flink master works perfectly fine with that version; 1.9
> will
> >>>>>>       likely do so too and 1.8 would require a minor adjustment.
> >>>>>>
> >>>>>>       Hence, there may be value in first doing a flink-shaded
> >>>>>> release so
> >>>>>>       we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 .
> >>>>>>
> >>>>>>
> >>>>>>       As for other jackson dependencies (coming from calcite, kafka,
> >>>>>>       kinesis), I ran the unit and end-to-end tests of master
> yesterday
> >>>>>>       will /all /jackson dependencies set to 2.10.1, and they
> passed. I
> >>>>>>       will open a PR soon-ish for making this change on master.
> >>>>>>
> >>>>>>       The question now is whether we want to backport this change to
> >>>>>>       1.8/1.9 .
> >>>>>>       Some code paths /may /not be covered by our tests, and
> transitive
> >>>>>>       jackson users /might /run into issues.
> >>>>>>       Alternatively, we could set this up as an opt-in upgrade, by
> >>>>>>       adding a separate profile that bumps the versions. This would
> >>>>>>       present users/providers who are concerned about the
> >>>>>>       vulnerabilities an easy workaround, at the risk of /some
> /things
> >>>>>>       /maybe /not working.
> >>>>>>
> >>>>>>       On 14/11/2019 03:16, Hequn Cheng wrote:
> >>>>>>>       Hi Chesnay, Jincheng
> >>>>>>>
> >>>>>>>       Sure, I think it's good to have these fixes.
> >>>>>>>       Thanks a lot for providing the information about the security
> >>>>>>>       vulnerabilities! @Chesnay
> >>>>>>>
> >>>>>>>       Best, Hequn
> >>>>>>>
> >>>>>>>       On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
> >>>> [hidden email]> <mailto:[hidden email]>
> >>>>>>>       wrote:
> >>>>>>>
> >>>>>>>>       +1 for try to eliminate the security vulnerabilities. Great
> >>>> thanks for
> >>>>>>>>       doing this important work, Chesnay!
> >>>>>>>>       What do you think Hequn ?
> >>>>>>>>
> >>>>>>>>       Best,
> >>>>>>>>       Jincheng
> >>>>>>>>
> >>>>>>>>       Chesnay Schepler<[hidden email]>
> >>>>>>>> <mailto:[hidden email]>
> >>>>    于2019年11月13日周三 下午5:17写道：
> >>>>>>>>>       It would be great if you could give me a day or 2 to check
> how
> >>>> easy it
> >>>>>>>>>       would be to bump the various jackson dependencies to
> >>>>>>>>> eliminate a
> >>>> few
> >>>>>>>>>       security vulnerabilities.
> >>>>>>>>>
> >>>>>>>>>       On 09/11/2019 05:10, jincheng sun wrote:
> >>>>>>>>>>       Hi Flink devs,
> >>>>>>>>>>
> >>>>>>>>>>       It has been more than 2 months since the 1.8.2 released.
> So,
> >>>> What do
> >>>>>>>>       you
> >>>>>>>>>>       think about releasing Flink 1.8.3 soon?
> >>>>>>>>>>
> >>>>>>>>>>       We already have many important bug fixes in the
> release-1.8
> >>>> branch (29
> >>>>>>>>>>       resolved issues).
> >>>>>>>>>>
> >>>>>>>>>>       Most notable fixes are:
> >>>>>>>>>>
> >>>>>>>>>>       - FLINK-14010 Dispatcher & JobManagers don't give up
> >>>>>>>>>> leadership
> >>>> when AM
> >>>>>>>>>       is
> >>>>>>>>>>       shut down
> >>>>>>>>>>       - FLINK-14315 NPE with JobMaster.disconnectTaskManager
> >>>>>>>>>>       - FLINK-12848 Method equals() in RowTypeInfo should
> consider
> >>>>>>>>       fieldsNames
> >>>>>>>>>>       - FLINK-12342 Yarn Resource Manager Acquires Too Many
> >>>>>>>>>> Containers
> >>>>>>>>>>       - FLINK-14589 Redundant slot requests with the same
> >>>> AllocationID leads
> >>>>>>>>       to
> >>>>>>>>>>       inconsistent slot table
> >>>>>>>>>>
> >>>>>>>>>>       Furthermore, the following critical issues is in progress,
> >>>> maybe we can
> >>>>>>>>>>       wait for it if it is not too much effort.
> >>>>>>>>>>
> >>>>>>>>>>       - FLINK-13184 Starting a TaskExecutor blocks the
> >>>> YarnResourceManager's
> >>>>>>>>>       main
> >>>>>>>>>>       thread
> >>>>>>>>>>
> >>>>>>>>>>       Please let me know what you think?
> >>>>>>>>>>
> >>>>>>>>>>       Best,
> >>>>>>>>>>       Jincheng
> >>>>>>>>>>
> >>>
> >>
>
>

[Chesnay Schepler-3]

I've kicked off a discussion about the next flink-shaded release, and
have opened PRs for adding the opt-in profile to 1.8/1.9.

On 15/11/2019 13:54, Hequn Cheng wrote:

> That's great, thank you very much! Ideally, we can kick off the release
> vote for the first RC of 1.8.3 within next week. :)
>
> On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <[hidden email]> wrote:
>
>> I'm not aware of any more planned changes to flink-shaded; so we could
>> start the release right away.
>>
>> On 15/11/2019 13:44, Hequn Cheng wrote:
>>> Hi,
>>>
>>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach for
>>> 1.8/1.9.
>>> @Ufuk Thank you for the nice summary.
>>>
>>> Looks good so far except that we need to postpone 1.8.3 a bit to first
>> do a
>>> flink-shaded release.
>>> BTW, @chesnay when would we plan to release the flink-shaded with
>> upgraded
>>> Jackson?
>>>
>>> Best, Hequn
>>>
>>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <[hidden email]>
>> wrote:
>>>> One small modification: the flink-shaded upgrade does not have to be
>>>> part of the profile; since it is only intended for internal use anyway
>>>> (and thus has limited exposure) we can be pretty sure this doesn't break
>>>> anything.
>>>>
>>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>>>>> Ufuk's summary is correct.
>>>>>
>>>>> There's a slight caveat in that we'd also have to bump the
>>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>>>>> but I have no concerns about this change.
>>>>>
>>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>>>>>> The opt-in approach seems reasonable to me. +1 to include the
>>>>>> profiles in
>>>>>> 1.8 and 1.9 without changing the default versions (including the
>> default
>>>>>> version of flink-shaded).
>>>>>>
>>>>>> As far as I can tell, the next steps would be:
>>>>>>
>>>>>> 1) Release flink-shaded with upgraded Jackson
>>>>>> 2a) Bump the flink-shaded version by default in master
>>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles
>>>>>> should also
>>>>>> cover the upgrade to the most recent flink-shaded version)
>>>>>>
>>>>>> @Chesnay: is this a correct summary?
>>>>>>
>>>>>> Note this would block the 1.8.3 release on step 1. As an upside, we
>>>>>> might
>>>>>> get some additional feedback until the 1.10 release with these
>>>>>> profiles in
>>>>>> case users make use of them with 1.8/1.9.
>>>>>>
>>>>>> – Ufuk
>>>>>>
>>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <[hidden email]
>>>>>> wrote:
>>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on master
>>>>>>> (and
>>>>>>> thus starting from 1.10.0) it's not opt-in.
>>>>>>>
>>>>>>> I have only proposed it as an opt-in because a) we usually do not
>> bump
>>>>>>> dependencies in bugfix releases and b) it's a short-term change that
>> we
>>>>>>> aren't allowing to mature properly.
>>>>>>> In contrast, the 1.10 release is significantly further away, hence no
>>>>>>> opt-in.
>>>>>>>
>>>>>>> Hence, I'm not concerned about such kind of ugprades being more
>> common
>>>>>>> in the future.
>>>>>>>
>>>>>>> We can certainly support every jackson version that fixes these
>>>>>>> vulnerabilities; individual modules can always use a different
>> version
>>>>>>> (that hopefully includes the fixes).
>>>>>>> Ideally of course we'd only be using 1 version, but that may or may
>> not
>>>>>>> be feasible.
>>>>>>>
>>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>>>>>>>> Hi Chesnay,
>>>>>>>>
>>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really a
>> good
>>>>>> job!
>>>>>>>> - Whether backport this change to 1.8/1.9
>>>>>>>> I had taken a quick look at the security vulnerabilities, some of
>> them
>>>>>>>> seem can lead to high-security problems, thus from my point of view,
>>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would like
>> to
>>>>>>>> trust your judgment as you are more professional at this problem.
>>>>>>>>
>>>>>>>> - How to port this change to 1.8/1.9
>>>>>>>> I think providing an opt-in upgrade is a good idea. Another question
>>>>>>>> here is whether do we plan to support multi jackson versions that
>> have
>>>>>>>> eliminated the security vulnerabilities. If we only plan to support
>>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an option,
>>>>>>>> users can downgrade the flink version if meet problems using the new
>>>>>>>> version. Of course, we will try our best to make the new release out
>>>>>>>> of question.
>>>>>>>> Another concern of making it an opt-in upgrade is, it will make our
>>>>>>>> build unlikely convergence as more and more build options will be
>>>>>>>> added when we upgrade a commonly used lib like this one.
>>>>>>>>
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>>> Best, Hequn
>>>>>>>>
>>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>> [hidden email]
>>>>>>>> <mailto:[hidden email]>> wrote:
>>>>>>>>
>>>>>>>>        So here's the state of things:
>>>>>>>>
>>>>>>>>
>>>>>>>>        The master of flink-shaded now uses jackson 2.10.1, which
>>>>>>>>        eliminates a whole category of security vulnerabilities.
>>>>>>>>        The flink master works perfectly fine with that version; 1.9
>> will
>>>>>>>>        likely do so too and 1.8 would require a minor adjustment.
>>>>>>>>
>>>>>>>>        Hence, there may be value in first doing a flink-shaded
>>>>>>>> release so
>>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 .
>>>>>>>>
>>>>>>>>
>>>>>>>>        As for other jackson dependencies (coming from calcite, kafka,
>>>>>>>>        kinesis), I ran the unit and end-to-end tests of master
>> yesterday
>>>>>>>>        will /all /jackson dependencies set to 2.10.1, and they
>> passed. I
>>>>>>>>        will open a PR soon-ish for making this change on master.
>>>>>>>>
>>>>>>>>        The question now is whether we want to backport this change to
>>>>>>>>        1.8/1.9 .
>>>>>>>>        Some code paths /may /not be covered by our tests, and
>> transitive
>>>>>>>>        jackson users /might /run into issues.
>>>>>>>>        Alternatively, we could set this up as an opt-in upgrade, by
>>>>>>>>        adding a separate profile that bumps the versions. This would
>>>>>>>>        present users/providers who are concerned about the
>>>>>>>>        vulnerabilities an easy workaround, at the risk of /some
>> /things
>>>>>>>>        /maybe /not working.
>>>>>>>>
>>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>>>>>>>>>        Hi Chesnay, Jincheng
>>>>>>>>>
>>>>>>>>>        Sure, I think it's good to have these fixes.
>>>>>>>>>        Thanks a lot for providing the information about the security
>>>>>>>>>        vulnerabilities! @Chesnay
>>>>>>>>>
>>>>>>>>>        Best, Hequn
>>>>>>>>>
>>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>>>>>> [hidden email]> <mailto:[hidden email]>
>>>>>>>>>        wrote:
>>>>>>>>>
>>>>>>>>>>        +1 for try to eliminate the security vulnerabilities. Great
>>>>>> thanks for
>>>>>>>>>>        doing this important work, Chesnay!
>>>>>>>>>>        What do you think Hequn ?
>>>>>>>>>>
>>>>>>>>>>        Best,
>>>>>>>>>>        Jincheng
>>>>>>>>>>
>>>>>>>>>>        Chesnay Schepler<[hidden email]>
>>>>>>>>>> <mailto:[hidden email]>
>>>>>>     于2019年11月13日周三 下午5:17写道：
>>>>>>>>>>>        It would be great if you could give me a day or 2 to check
>> how
>>>>>> easy it
>>>>>>>>>>>        would be to bump the various jackson dependencies to
>>>>>>>>>>> eliminate a
>>>>>> few
>>>>>>>>>>>        security vulnerabilities.
>>>>>>>>>>>
>>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>>>>>>>>>>>>        Hi Flink devs,
>>>>>>>>>>>>
>>>>>>>>>>>>        It has been more than 2 months since the 1.8.2 released.
>> So,
>>>>>> What do
>>>>>>>>>>        you
>>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>>>>>>>>>>>>
>>>>>>>>>>>>        We already have many important bug fixes in the
>> release-1.8
>>>>>> branch (29
>>>>>>>>>>>>        resolved issues).
>>>>>>>>>>>>
>>>>>>>>>>>>        Most notable fixes are:
>>>>>>>>>>>>
>>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't give up
>>>>>>>>>>>> leadership
>>>>>> when AM
>>>>>>>>>>>        is
>>>>>>>>>>>>        shut down
>>>>>>>>>>>>        - FLINK-14315 NPE with JobMaster.disconnectTaskManager
>>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo should
>> consider
>>>>>>>>>>        fieldsNames
>>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too Many
>>>>>>>>>>>> Containers
>>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the same
>>>>>> AllocationID leads
>>>>>>>>>>        to
>>>>>>>>>>>>        inconsistent slot table
>>>>>>>>>>>>
>>>>>>>>>>>>        Furthermore, the following critical issues is in progress,
>>>>>> maybe we can
>>>>>>>>>>>>        wait for it if it is not too much effort.
>>>>>>>>>>>>
>>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
>>>>>> YarnResourceManager's
>>>>>>>>>>>        main
>>>>>>>>>>>>        thread
>>>>>>>>>>>>
>>>>>>>>>>>>        Please let me know what you think?
>>>>>>>>>>>>
>>>>>>>>>>>>        Best,
>>>>>>>>>>>>        Jincheng
>>>>>>>>>>>>
>>

[Ufuk Celebi-2]

Thanks Chesnay.

I'm also +1 to release 1.8.3 asap without the changes for the Jackson
version bump and leave those for a future release. Realistically, the
flink-shaded release will take until mid next week or end of next week. But
please correct me if you think that it should not take that long or it's OK
to block the 1.8.3 release on the flink-shaded release.

– Ufuk


On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]> wrote:

> I've kicked off a discussion about the next flink-shaded release, and
> have opened PRs for adding the opt-in profile to 1.8/1.9.
>
> On 15/11/2019 13:54, Hequn Cheng wrote:
> > That's great, thank you very much! Ideally, we can kick off the release
> > vote for the first RC of 1.8.3 within next week. :)
> >
> > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <[hidden email]>
> wrote:
> >
> >> I'm not aware of any more planned changes to flink-shaded; so we could
> >> start the release right away.
> >>
> >> On 15/11/2019 13:44, Hequn Cheng wrote:
> >>> Hi,
> >>>
> >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach
> for
> >>> 1.8/1.9.
> >>> @Ufuk Thank you for the nice summary.
> >>>
> >>> Looks good so far except that we need to postpone 1.8.3 a bit to first
> >> do a
> >>> flink-shaded release.
> >>> BTW, @chesnay when would we plan to release the flink-shaded with
> >> upgraded
> >>> Jackson?
> >>>
> >>> Best, Hequn
> >>>
> >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <[hidden email]>
> >> wrote:
> >>>> One small modification: the flink-shaded upgrade does not have to be
> >>>> part of the profile; since it is only intended for internal use anyway
> >>>> (and thus has limited exposure) we can be pretty sure this doesn't
> break
> >>>> anything.
> >>>>
> >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
> >>>>> Ufuk's summary is correct.
> >>>>>
> >>>>> There's a slight caveat in that we'd also have to bump the
> >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
> >>>>> but I have no concerns about this change.
> >>>>>
> >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
> >>>>>> The opt-in approach seems reasonable to me. +1 to include the
> >>>>>> profiles in
> >>>>>> 1.8 and 1.9 without changing the default versions (including the
> >> default
> >>>>>> version of flink-shaded).
> >>>>>>
> >>>>>> As far as I can tell, the next steps would be:
> >>>>>>
> >>>>>> 1) Release flink-shaded with upgraded Jackson
> >>>>>> 2a) Bump the flink-shaded version by default in master
> >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles
> >>>>>> should also
> >>>>>> cover the upgrade to the most recent flink-shaded version)
> >>>>>>
> >>>>>> @Chesnay: is this a correct summary?
> >>>>>>
> >>>>>> Note this would block the 1.8.3 release on step 1. As an upside, we
> >>>>>> might
> >>>>>> get some additional feedback until the 1.10 release with these
> >>>>>> profiles in
> >>>>>> case users make use of them with 1.8/1.9.
> >>>>>>
> >>>>>> – Ufuk
> >>>>>>
> >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
> [hidden email]
> >>>>>> wrote:
> >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on master
> >>>>>>> (and
> >>>>>>> thus starting from 1.10.0) it's not opt-in.
> >>>>>>>
> >>>>>>> I have only proposed it as an opt-in because a) we usually do not
> >> bump
> >>>>>>> dependencies in bugfix releases and b) it's a short-term change
> that
> >> we
> >>>>>>> aren't allowing to mature properly.
> >>>>>>> In contrast, the 1.10 release is significantly further away, hence
> no
> >>>>>>> opt-in.
> >>>>>>>
> >>>>>>> Hence, I'm not concerned about such kind of ugprades being more
> >> common
> >>>>>>> in the future.
> >>>>>>>
> >>>>>>> We can certainly support every jackson version that fixes these
> >>>>>>> vulnerabilities; individual modules can always use a different
> >> version
> >>>>>>> (that hopefully includes the fixes).
> >>>>>>> Ideally of course we'd only be using 1 version, but that may or may
> >> not
> >>>>>>> be feasible.
> >>>>>>>
> >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
> >>>>>>>> Hi Chesnay,
> >>>>>>>>
> >>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really a
> >> good
> >>>>>> job!
> >>>>>>>> - Whether backport this change to 1.8/1.9
> >>>>>>>> I had taken a quick look at the security vulnerabilities, some of
> >> them
> >>>>>>>> seem can lead to high-security problems, thus from my point of
> view,
> >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would like
> >> to
> >>>>>>>> trust your judgment as you are more professional at this problem.
> >>>>>>>>
> >>>>>>>> - How to port this change to 1.8/1.9
> >>>>>>>> I think providing an opt-in upgrade is a good idea. Another
> question
> >>>>>>>> here is whether do we plan to support multi jackson versions that
> >> have
> >>>>>>>> eliminated the security vulnerabilities. If we only plan to
> support
> >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an
> option,
> >>>>>>>> users can downgrade the flink version if meet problems using the
> new
> >>>>>>>> version. Of course, we will try our best to make the new release
> out
> >>>>>>>> of question.
> >>>>>>>> Another concern of making it an opt-in upgrade is, it will make
> our
> >>>>>>>> build unlikely convergence as more and more build options will be
> >>>>>>>> added when we upgrade a commonly used lib like this one.
> >>>>>>>>
> >>>>>>>> What do you think?
> >>>>>>>>
> >>>>>>>> Best, Hequn
> >>>>>>>>
> >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
> >> [hidden email]
> >>>>>>>> <mailto:[hidden email]>> wrote:
> >>>>>>>>
> >>>>>>>>        So here's the state of things:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1, which
> >>>>>>>>        eliminates a whole category of security vulnerabilities.
> >>>>>>>>        The flink master works perfectly fine with that version;
> 1.9
> >> will
> >>>>>>>>        likely do so too and 1.8 would require a minor adjustment.
> >>>>>>>>
> >>>>>>>>        Hence, there may be value in first doing a flink-shaded
> >>>>>>>> release so
> >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 .
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>        As for other jackson dependencies (coming from calcite,
> kafka,
> >>>>>>>>        kinesis), I ran the unit and end-to-end tests of master
> >> yesterday
> >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and they
> >> passed. I
> >>>>>>>>        will open a PR soon-ish for making this change on master.
> >>>>>>>>
> >>>>>>>>        The question now is whether we want to backport this
> change to
> >>>>>>>>        1.8/1.9 .
> >>>>>>>>        Some code paths /may /not be covered by our tests, and
> >> transitive
> >>>>>>>>        jackson users /might /run into issues.
> >>>>>>>>        Alternatively, we could set this up as an opt-in upgrade,
> by
> >>>>>>>>        adding a separate profile that bumps the versions. This
> would
> >>>>>>>>        present users/providers who are concerned about the
> >>>>>>>>        vulnerabilities an easy workaround, at the risk of /some
> >> /things
> >>>>>>>>        /maybe /not working.
> >>>>>>>>
> >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
> >>>>>>>>>        Hi Chesnay, Jincheng
> >>>>>>>>>
> >>>>>>>>>        Sure, I think it's good to have these fixes.
> >>>>>>>>>        Thanks a lot for providing the information about the
> security
> >>>>>>>>>        vulnerabilities! @Chesnay
> >>>>>>>>>
> >>>>>>>>>        Best, Hequn
> >>>>>>>>>
> >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
> >>>>>> [hidden email]> <mailto:[hidden email]>
> >>>>>>>>>        wrote:
> >>>>>>>>>
> >>>>>>>>>>        +1 for try to eliminate the security vulnerabilities.
> Great
> >>>>>> thanks for
> >>>>>>>>>>        doing this important work, Chesnay!
> >>>>>>>>>>        What do you think Hequn ?
> >>>>>>>>>>
> >>>>>>>>>>        Best,
> >>>>>>>>>>        Jincheng
> >>>>>>>>>>
> >>>>>>>>>>        Chesnay Schepler<[hidden email]>
> >>>>>>>>>> <mailto:[hidden email]>
> >>>>>>     于2019年11月13日周三 下午5:17写道：
> >>>>>>>>>>>        It would be great if you could give me a day or 2 to
> check
> >> how
> >>>>>> easy it
> >>>>>>>>>>>        would be to bump the various jackson dependencies to
> >>>>>>>>>>> eliminate a
> >>>>>> few
> >>>>>>>>>>>        security vulnerabilities.
> >>>>>>>>>>>
> >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
> >>>>>>>>>>>>        Hi Flink devs,
> >>>>>>>>>>>>
> >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
> released.
> >> So,
> >>>>>> What do
> >>>>>>>>>>        you
> >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
> >>>>>>>>>>>>
> >>>>>>>>>>>>        We already have many important bug fixes in the
> >> release-1.8
> >>>>>> branch (29
> >>>>>>>>>>>>        resolved issues).
> >>>>>>>>>>>>
> >>>>>>>>>>>>        Most notable fixes are:
> >>>>>>>>>>>>
> >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't give up
> >>>>>>>>>>>> leadership
> >>>>>> when AM
> >>>>>>>>>>>        is
> >>>>>>>>>>>>        shut down
> >>>>>>>>>>>>        - FLINK-14315 NPE with JobMaster.disconnectTaskManager
> >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo should
> >> consider
> >>>>>>>>>>        fieldsNames
> >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too Many
> >>>>>>>>>>>> Containers
> >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the same
> >>>>>> AllocationID leads
> >>>>>>>>>>        to
> >>>>>>>>>>>>        inconsistent slot table
> >>>>>>>>>>>>
> >>>>>>>>>>>>        Furthermore, the following critical issues is in
> progress,
> >>>>>> maybe we can
> >>>>>>>>>>>>        wait for it if it is not too much effort.
> >>>>>>>>>>>>
> >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
> >>>>>> YarnResourceManager's
> >>>>>>>>>>>        main
> >>>>>>>>>>>>        thread
> >>>>>>>>>>>>
> >>>>>>>>>>>>        Please let me know what you think?
> >>>>>>>>>>>>
> >>>>>>>>>>>>        Best,
> >>>>>>>>>>>>        Jincheng
> >>>>>>>>>>>>
> >>
>
>

[Zhu Zhu]

Hi Jincheng & Hequn

Thanks for driving the releasing of 1.8.3.

I am now working on FLINK-14735. The fix avoids duplicated input
checking when scheduling ALL-to-ALL
connected downstream consumers with ALL input constraints. The duplicated
checking can cause severe
performance issues for large scale jobs. So I hope the fix could be
released with 1.8.3.

The fix is already merged into master, and is now in the process of
backporting to 1.8.

Thanks,
Zhu Zhu

Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：

> Thanks Chesnay.
>
> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
> version bump and leave those for a future release. Realistically, the
> flink-shaded release will take until mid next week or end of next week. But
> please correct me if you think that it should not take that long or it's OK
> to block the 1.8.3 release on the flink-shaded release.
>
> – Ufuk
>
>
> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]>
> wrote:
>
> > I've kicked off a discussion about the next flink-shaded release, and
> > have opened PRs for adding the opt-in profile to 1.8/1.9.
> >
> > On 15/11/2019 13:54, Hequn Cheng wrote:
> > > That's great, thank you very much! Ideally, we can kick off the release
> > > vote for the first RC of 1.8.3 within next week. :)
> > >
> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <[hidden email]>
> > wrote:
> > >
> > >> I'm not aware of any more planned changes to flink-shaded; so we could
> > >> start the release right away.
> > >>
> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
> > >>> Hi,
> > >>>
> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach
> > for
> > >>> 1.8/1.9.
> > >>> @Ufuk Thank you for the nice summary.
> > >>>
> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
> first
> > >> do a
> > >>> flink-shaded release.
> > >>> BTW, @chesnay when would we plan to release the flink-shaded with
> > >> upgraded
> > >>> Jackson?
> > >>>
> > >>> Best, Hequn
> > >>>
> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <[hidden email]
> >
> > >> wrote:
> > >>>> One small modification: the flink-shaded upgrade does not have to be
> > >>>> part of the profile; since it is only intended for internal use
> anyway
> > >>>> (and thus has limited exposure) we can be pretty sure this doesn't
> > break
> > >>>> anything.
> > >>>>
> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
> > >>>>> Ufuk's summary is correct.
> > >>>>>
> > >>>>> There's a slight caveat in that we'd also have to bump the
> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
> > >>>>> but I have no concerns about this change.
> > >>>>>
> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the
> > >>>>>> profiles in
> > >>>>>> 1.8 and 1.9 without changing the default versions (including the
> > >> default
> > >>>>>> version of flink-shaded).
> > >>>>>>
> > >>>>>> As far as I can tell, the next steps would be:
> > >>>>>>
> > >>>>>> 1) Release flink-shaded with upgraded Jackson
> > >>>>>> 2a) Bump the flink-shaded version by default in master
> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles
> > >>>>>> should also
> > >>>>>> cover the upgrade to the most recent flink-shaded version)
> > >>>>>>
> > >>>>>> @Chesnay: is this a correct summary?
> > >>>>>>
> > >>>>>> Note this would block the 1.8.3 release on step 1. As an upside,
> we
> > >>>>>> might
> > >>>>>> get some additional feedback until the 1.10 release with these
> > >>>>>> profiles in
> > >>>>>> case users make use of them with 1.8/1.9.
> > >>>>>>
> > >>>>>> – Ufuk
> > >>>>>>
> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
> > [hidden email]
> > >>>>>> wrote:
> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on
> master
> > >>>>>>> (and
> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
> > >>>>>>>
> > >>>>>>> I have only proposed it as an opt-in because a) we usually do not
> > >> bump
> > >>>>>>> dependencies in bugfix releases and b) it's a short-term change
> > that
> > >> we
> > >>>>>>> aren't allowing to mature properly.
> > >>>>>>> In contrast, the 1.10 release is significantly further away,
> hence
> > no
> > >>>>>>> opt-in.
> > >>>>>>>
> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more
> > >> common
> > >>>>>>> in the future.
> > >>>>>>>
> > >>>>>>> We can certainly support every jackson version that fixes these
> > >>>>>>> vulnerabilities; individual modules can always use a different
> > >> version
> > >>>>>>> (that hopefully includes the fixes).
> > >>>>>>> Ideally of course we'd only be using 1 version, but that may or
> may
> > >> not
> > >>>>>>> be feasible.
> > >>>>>>>
> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
> > >>>>>>>> Hi Chesnay,
> > >>>>>>>>
> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really a
> > >> good
> > >>>>>> job!
> > >>>>>>>> - Whether backport this change to 1.8/1.9
> > >>>>>>>> I had taken a quick look at the security vulnerabilities, some
> of
> > >> them
> > >>>>>>>> seem can lead to high-security problems, thus from my point of
> > view,
> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would
> like
> > >> to
> > >>>>>>>> trust your judgment as you are more professional at this
> problem.
> > >>>>>>>>
> > >>>>>>>> - How to port this change to 1.8/1.9
> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another
> > question
> > >>>>>>>> here is whether do we plan to support multi jackson versions
> that
> > >> have
> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to
> > support
> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an
> > option,
> > >>>>>>>> users can downgrade the flink version if meet problems using the
> > new
> > >>>>>>>> version. Of course, we will try our best to make the new release
> > out
> > >>>>>>>> of question.
> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will make
> > our
> > >>>>>>>> build unlikely convergence as more and more build options will
> be
> > >>>>>>>> added when we upgrade a commonly used lib like this one.
> > >>>>>>>>
> > >>>>>>>> What do you think?
> > >>>>>>>>
> > >>>>>>>> Best, Hequn
> > >>>>>>>>
> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
> > >> [hidden email]
> > >>>>>>>> <mailto:[hidden email]>> wrote:
> > >>>>>>>>
> > >>>>>>>>        So here's the state of things:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1, which
> > >>>>>>>>        eliminates a whole category of security vulnerabilities.
> > >>>>>>>>        The flink master works perfectly fine with that version;
> > 1.9
> > >> will
> > >>>>>>>>        likely do so too and 1.8 would require a minor
> adjustment.
> > >>>>>>>>
> > >>>>>>>>        Hence, there may be value in first doing a flink-shaded
> > >>>>>>>> release so
> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and
> 1.9.2 .
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>        As for other jackson dependencies (coming from calcite,
> > kafka,
> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of master
> > >> yesterday
> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and they
> > >> passed. I
> > >>>>>>>>        will open a PR soon-ish for making this change on master.
> > >>>>>>>>
> > >>>>>>>>        The question now is whether we want to backport this
> > change to
> > >>>>>>>>        1.8/1.9 .
> > >>>>>>>>        Some code paths /may /not be covered by our tests, and
> > >> transitive
> > >>>>>>>>        jackson users /might /run into issues.
> > >>>>>>>>        Alternatively, we could set this up as an opt-in upgrade,
> > by
> > >>>>>>>>        adding a separate profile that bumps the versions. This
> > would
> > >>>>>>>>        present users/providers who are concerned about the
> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of /some
> > >> /things
> > >>>>>>>>        /maybe /not working.
> > >>>>>>>>
> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
> > >>>>>>>>>        Hi Chesnay, Jincheng
> > >>>>>>>>>
> > >>>>>>>>>        Sure, I think it's good to have these fixes.
> > >>>>>>>>>        Thanks a lot for providing the information about the
> > security
> > >>>>>>>>>        vulnerabilities! @Chesnay
> > >>>>>>>>>
> > >>>>>>>>>        Best, Hequn
> > >>>>>>>>>
> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
> > >>>>>> [hidden email]> <mailto:[hidden email]>
> > >>>>>>>>>        wrote:
> > >>>>>>>>>
> > >>>>>>>>>>        +1 for try to eliminate the security vulnerabilities.
> > Great
> > >>>>>> thanks for
> > >>>>>>>>>>        doing this important work, Chesnay!
> > >>>>>>>>>>        What do you think Hequn ?
> > >>>>>>>>>>
> > >>>>>>>>>>        Best,
> > >>>>>>>>>>        Jincheng
> > >>>>>>>>>>
> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
> > >>>>>>>>>> <mailto:[hidden email]>
> > >>>>>>     于2019年11月13日周三 下午5:17写道：
> > >>>>>>>>>>>        It would be great if you could give me a day or 2 to
> > check
> > >> how
> > >>>>>> easy it
> > >>>>>>>>>>>        would be to bump the various jackson dependencies to
> > >>>>>>>>>>> eliminate a
> > >>>>>> few
> > >>>>>>>>>>>        security vulnerabilities.
> > >>>>>>>>>>>
> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
> > >>>>>>>>>>>>        Hi Flink devs,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
> > released.
> > >> So,
> > >>>>>> What do
> > >>>>>>>>>>        you
> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        We already have many important bug fixes in the
> > >> release-1.8
> > >>>>>> branch (29
> > >>>>>>>>>>>>        resolved issues).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        Most notable fixes are:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't give up
> > >>>>>>>>>>>> leadership
> > >>>>>> when AM
> > >>>>>>>>>>>        is
> > >>>>>>>>>>>>        shut down
> > >>>>>>>>>>>>        - FLINK-14315 NPE with
> JobMaster.disconnectTaskManager
> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo should
> > >> consider
> > >>>>>>>>>>        fieldsNames
> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too Many
> > >>>>>>>>>>>> Containers
> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the same
> > >>>>>> AllocationID leads
> > >>>>>>>>>>        to
> > >>>>>>>>>>>>        inconsistent slot table
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        Furthermore, the following critical issues is in
> > progress,
> > >>>>>> maybe we can
> > >>>>>>>>>>>>        wait for it if it is not too much effort.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
> > >>>>>> YarnResourceManager's
> > >>>>>>>>>>>        main
> > >>>>>>>>>>>>        thread
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        Please let me know what you think?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>        Best,
> > >>>>>>>>>>>>        Jincheng
> > >>>>>>>>>>>>
> > >>
> >
> >
>

[Hequn Cheng]

Hi Zhu Zhu,

Thanks a lot for letting us know!
We can't cut the first RC right now due to the wait of the flink-shade
release, so go ahead.

Theoretically, we will cut the first RC of 1.8.3 and vote for it once the
release of flink-shade is done,
but I will try my best to have it in 1.8.3. Hope we can get it on board on
time. :)

Best, Hequn

On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:

> Hi Jincheng & Hequn
>
> Thanks for driving the releasing of 1.8.3.
>
> I am now working on FLINK-14735. The fix avoids duplicated input
> checking when scheduling ALL-to-ALL
> connected downstream consumers with ALL input constraints. The duplicated
> checking can cause severe
> performance issues for large scale jobs. So I hope the fix could be
> released with 1.8.3.
>
> The fix is already merged into master, and is now in the process of
> backporting to 1.8.
>
> Thanks,
> Zhu Zhu
>
> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>
>> Thanks Chesnay.
>>
>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
>> version bump and leave those for a future release. Realistically, the
>> flink-shaded release will take until mid next week or end of next week.
>> But
>> please correct me if you think that it should not take that long or it's
>> OK
>> to block the 1.8.3 release on the flink-shaded release.
>>
>> – Ufuk
>>
>>
>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]>
>> wrote:
>>
>> > I've kicked off a discussion about the next flink-shaded release, and
>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>> >
>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>> > > That's great, thank you very much! Ideally, we can kick off the
>> release
>> > > vote for the first RC of 1.8.3 within next week. :)
>> > >
>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <[hidden email]>
>> > wrote:
>> > >
>> > >> I'm not aware of any more planned changes to flink-shaded; so we
>> could
>> > >> start the release right away.
>> > >>
>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>> > >>> Hi,
>> > >>>
>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach
>> > for
>> > >>> 1.8/1.9.
>> > >>> @Ufuk Thank you for the nice summary.
>> > >>>
>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
>> first
>> > >> do a
>> > >>> flink-shaded release.
>> > >>> BTW, @chesnay when would we plan to release the flink-shaded with
>> > >> upgraded
>> > >>> Jackson?
>> > >>>
>> > >>> Best, Hequn
>> > >>>
>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>> [hidden email]>
>> > >> wrote:
>> > >>>> One small modification: the flink-shaded upgrade does not have to
>> be
>> > >>>> part of the profile; since it is only intended for internal use
>> anyway
>> > >>>> (and thus has limited exposure) we can be pretty sure this doesn't
>> > break
>> > >>>> anything.
>> > >>>>
>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>> > >>>>> Ufuk's summary is correct.
>> > >>>>>
>> > >>>>> There's a slight caveat in that we'd also have to bump the
>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>> > >>>>> but I have no concerns about this change.
>> > >>>>>
>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the
>> > >>>>>> profiles in
>> > >>>>>> 1.8 and 1.9 without changing the default versions (including the
>> > >> default
>> > >>>>>> version of flink-shaded).
>> > >>>>>>
>> > >>>>>> As far as I can tell, the next steps would be:
>> > >>>>>>
>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles
>> > >>>>>> should also
>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
>> > >>>>>>
>> > >>>>>> @Chesnay: is this a correct summary?
>> > >>>>>>
>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an upside,
>> we
>> > >>>>>> might
>> > >>>>>> get some additional feedback until the 1.10 release with these
>> > >>>>>> profiles in
>> > >>>>>> case users make use of them with 1.8/1.9.
>> > >>>>>>
>> > >>>>>> – Ufuk
>> > >>>>>>
>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>> > [hidden email]
>> > >>>>>> wrote:
>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on
>> master
>> > >>>>>>> (and
>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>> > >>>>>>>
>> > >>>>>>> I have only proposed it as an opt-in because a) we usually do
>> not
>> > >> bump
>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term change
>> > that
>> > >> we
>> > >>>>>>> aren't allowing to mature properly.
>> > >>>>>>> In contrast, the 1.10 release is significantly further away,
>> hence
>> > no
>> > >>>>>>> opt-in.
>> > >>>>>>>
>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more
>> > >> common
>> > >>>>>>> in the future.
>> > >>>>>>>
>> > >>>>>>> We can certainly support every jackson version that fixes these
>> > >>>>>>> vulnerabilities; individual modules can always use a different
>> > >> version
>> > >>>>>>> (that hopefully includes the fixes).
>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may or
>> may
>> > >> not
>> > >>>>>>> be feasible.
>> > >>>>>>>
>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>> > >>>>>>>> Hi Chesnay,
>> > >>>>>>>>
>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. Really
>> a
>> > >> good
>> > >>>>>> job!
>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>> > >>>>>>>> I had taken a quick look at the security vulnerabilities, some
>> of
>> > >> them
>> > >>>>>>>> seem can lead to high-security problems, thus from my point of
>> > view,
>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would
>> like
>> > >> to
>> > >>>>>>>> trust your judgment as you are more professional at this
>> problem.
>> > >>>>>>>>
>> > >>>>>>>> - How to port this change to 1.8/1.9
>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another
>> > question
>> > >>>>>>>> here is whether do we plan to support multi jackson versions
>> that
>> > >> have
>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to
>> > support
>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an
>> > option,
>> > >>>>>>>> users can downgrade the flink version if meet problems using
>> the
>> > new
>> > >>>>>>>> version. Of course, we will try our best to make the new
>> release
>> > out
>> > >>>>>>>> of question.
>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will make
>> > our
>> > >>>>>>>> build unlikely convergence as more and more build options will
>> be
>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
>> > >>>>>>>>
>> > >>>>>>>> What do you think?
>> > >>>>>>>>
>> > >>>>>>>> Best, Hequn
>> > >>>>>>>>
>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>> > >> [hidden email]
>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>        So here's the state of things:
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1,
>> which
>> > >>>>>>>>        eliminates a whole category of security vulnerabilities.
>> > >>>>>>>>        The flink master works perfectly fine with that version;
>> > 1.9
>> > >> will
>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>> adjustment.
>> > >>>>>>>>
>> > >>>>>>>>        Hence, there may be value in first doing a flink-shaded
>> > >>>>>>>> release so
>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and
>> 1.9.2 .
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>        As for other jackson dependencies (coming from calcite,
>> > kafka,
>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of master
>> > >> yesterday
>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and they
>> > >> passed. I
>> > >>>>>>>>        will open a PR soon-ish for making this change on
>> master.
>> > >>>>>>>>
>> > >>>>>>>>        The question now is whether we want to backport this
>> > change to
>> > >>>>>>>>        1.8/1.9 .
>> > >>>>>>>>        Some code paths /may /not be covered by our tests, and
>> > >> transitive
>> > >>>>>>>>        jackson users /might /run into issues.
>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>> upgrade,
>> > by
>> > >>>>>>>>        adding a separate profile that bumps the versions. This
>> > would
>> > >>>>>>>>        present users/providers who are concerned about the
>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of /some
>> > >> /things
>> > >>>>>>>>        /maybe /not working.
>> > >>>>>>>>
>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>> > >>>>>>>>>        Hi Chesnay, Jincheng
>> > >>>>>>>>>
>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>> > >>>>>>>>>        Thanks a lot for providing the information about the
>> > security
>> > >>>>>>>>>        vulnerabilities! @Chesnay
>> > >>>>>>>>>
>> > >>>>>>>>>        Best, Hequn
>> > >>>>>>>>>
>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>> > >>>>>> [hidden email]> <mailto:[hidden email]>
>> > >>>>>>>>>        wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>>        +1 for try to eliminate the security vulnerabilities.
>> > Great
>> > >>>>>> thanks for
>> > >>>>>>>>>>        doing this important work, Chesnay!
>> > >>>>>>>>>>        What do you think Hequn ?
>> > >>>>>>>>>>
>> > >>>>>>>>>>        Best,
>> > >>>>>>>>>>        Jincheng
>> > >>>>>>>>>>
>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>> > >>>>>>>>>> <mailto:[hidden email]>
>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>> > >>>>>>>>>>>        It would be great if you could give me a day or 2 to
>> > check
>> > >> how
>> > >>>>>> easy it
>> > >>>>>>>>>>>        would be to bump the various jackson dependencies to
>> > >>>>>>>>>>> eliminate a
>> > >>>>>> few
>> > >>>>>>>>>>>        security vulnerabilities.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>> > >>>>>>>>>>>>        Hi Flink devs,
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
>> > released.
>> > >> So,
>> > >>>>>> What do
>> > >>>>>>>>>>        you
>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        We already have many important bug fixes in the
>> > >> release-1.8
>> > >>>>>> branch (29
>> > >>>>>>>>>>>>        resolved issues).
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        Most notable fixes are:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't give up
>> > >>>>>>>>>>>> leadership
>> > >>>>>> when AM
>> > >>>>>>>>>>>        is
>> > >>>>>>>>>>>>        shut down
>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>> JobMaster.disconnectTaskManager
>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo should
>> > >> consider
>> > >>>>>>>>>>        fieldsNames
>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too
>> Many
>> > >>>>>>>>>>>> Containers
>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the same
>> > >>>>>> AllocationID leads
>> > >>>>>>>>>>        to
>> > >>>>>>>>>>>>        inconsistent slot table
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        Furthermore, the following critical issues is in
>> > progress,
>> > >>>>>> maybe we can
>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
>> > >>>>>> YarnResourceManager's
>> > >>>>>>>>>>>        main
>> > >>>>>>>>>>>>        thread
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        Please let me know what you think?
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>        Best,
>> > >>>>>>>>>>>>        Jincheng
>> > >>>>>>>>>>>>
>> > >>
>> >
>> >
>>
>

[Zhu Zhu]

Hi Hequn,

Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
Given that this fix is for batch job only and batch is not very good in
1.8, I think it is a not blocker of release 1.8.3.
So just don't be blocked by it and feel free to cut the RC when other
blocking issues are resolved.

Thanks,
Zhu Zhu

Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：

> Hi Zhu Zhu,
>
> Thanks a lot for letting us know!
> We can't cut the first RC right now due to the wait of the flink-shade
> release, so go ahead.
>
> Theoretically, we will cut the first RC of 1.8.3 and vote for it once the
> release of flink-shade is done,
> but I will try my best to have it in 1.8.3. Hope we can get it on board on
> time. :)
>
> Best, Hequn
>
> On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
>
>> Hi Jincheng & Hequn
>>
>> Thanks for driving the releasing of 1.8.3.
>>
>> I am now working on FLINK-14735. The fix avoids duplicated input
>> checking when scheduling ALL-to-ALL
>> connected downstream consumers with ALL input constraints. The duplicated
>> checking can cause severe
>> performance issues for large scale jobs. So I hope the fix could be
>> released with 1.8.3.
>>
>> The fix is already merged into master, and is now in the process of
>> backporting to 1.8.
>>
>> Thanks,
>> Zhu Zhu
>>
>> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>>
>>> Thanks Chesnay.
>>>
>>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
>>> version bump and leave those for a future release. Realistically, the
>>> flink-shaded release will take until mid next week or end of next week.
>>> But
>>> please correct me if you think that it should not take that long or it's
>>> OK
>>> to block the 1.8.3 release on the flink-shaded release.
>>>
>>> – Ufuk
>>>
>>>
>>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]>
>>> wrote:
>>>
>>> > I've kicked off a discussion about the next flink-shaded release, and
>>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>>> >
>>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>>> > > That's great, thank you very much! Ideally, we can kick off the
>>> release
>>> > > vote for the first RC of 1.8.3 within next week. :)
>>> > >
>>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <[hidden email]
>>> >
>>> > wrote:
>>> > >
>>> > >> I'm not aware of any more planned changes to flink-shaded; so we
>>> could
>>> > >> start the release right away.
>>> > >>
>>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>>> > >>> Hi,
>>> > >>>
>>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>>> approach
>>> > for
>>> > >>> 1.8/1.9.
>>> > >>> @Ufuk Thank you for the nice summary.
>>> > >>>
>>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
>>> first
>>> > >> do a
>>> > >>> flink-shaded release.
>>> > >>> BTW, @chesnay when would we plan to release the flink-shaded with
>>> > >> upgraded
>>> > >>> Jackson?
>>> > >>>
>>> > >>> Best, Hequn
>>> > >>>
>>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>>> [hidden email]>
>>> > >> wrote:
>>> > >>>> One small modification: the flink-shaded upgrade does not have to
>>> be
>>> > >>>> part of the profile; since it is only intended for internal use
>>> anyway
>>> > >>>> (and thus has limited exposure) we can be pretty sure this doesn't
>>> > break
>>> > >>>> anything.
>>> > >>>>
>>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>>> > >>>>> Ufuk's summary is correct.
>>> > >>>>>
>>> > >>>>> There's a slight caveat in that we'd also have to bump the
>>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>>> > >>>>> but I have no concerns about this change.
>>> > >>>>>
>>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the
>>> > >>>>>> profiles in
>>> > >>>>>> 1.8 and 1.9 without changing the default versions (including the
>>> > >> default
>>> > >>>>>> version of flink-shaded).
>>> > >>>>>>
>>> > >>>>>> As far as I can tell, the next steps would be:
>>> > >>>>>>
>>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles
>>> > >>>>>> should also
>>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
>>> > >>>>>>
>>> > >>>>>> @Chesnay: is this a correct summary?
>>> > >>>>>>
>>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
>>> upside, we
>>> > >>>>>> might
>>> > >>>>>> get some additional feedback until the 1.10 release with these
>>> > >>>>>> profiles in
>>> > >>>>>> case users make use of them with 1.8/1.9.
>>> > >>>>>>
>>> > >>>>>> – Ufuk
>>> > >>>>>>
>>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>>> > [hidden email]
>>> > >>>>>> wrote:
>>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on
>>> master
>>> > >>>>>>> (and
>>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>>> > >>>>>>>
>>> > >>>>>>> I have only proposed it as an opt-in because a) we usually do
>>> not
>>> > >> bump
>>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term change
>>> > that
>>> > >> we
>>> > >>>>>>> aren't allowing to mature properly.
>>> > >>>>>>> In contrast, the 1.10 release is significantly further away,
>>> hence
>>> > no
>>> > >>>>>>> opt-in.
>>> > >>>>>>>
>>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more
>>> > >> common
>>> > >>>>>>> in the future.
>>> > >>>>>>>
>>> > >>>>>>> We can certainly support every jackson version that fixes these
>>> > >>>>>>> vulnerabilities; individual modules can always use a different
>>> > >> version
>>> > >>>>>>> (that hopefully includes the fixes).
>>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may
>>> or may
>>> > >> not
>>> > >>>>>>> be feasible.
>>> > >>>>>>>
>>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>>> > >>>>>>>> Hi Chesnay,
>>> > >>>>>>>>
>>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
>>> Really a
>>> > >> good
>>> > >>>>>> job!
>>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>>> > >>>>>>>> I had taken a quick look at the security vulnerabilities,
>>> some of
>>> > >> them
>>> > >>>>>>>> seem can lead to high-security problems, thus from my point of
>>> > view,
>>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would
>>> like
>>> > >> to
>>> > >>>>>>>> trust your judgment as you are more professional at this
>>> problem.
>>> > >>>>>>>>
>>> > >>>>>>>> - How to port this change to 1.8/1.9
>>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another
>>> > question
>>> > >>>>>>>> here is whether do we plan to support multi jackson versions
>>> that
>>> > >> have
>>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to
>>> > support
>>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an
>>> > option,
>>> > >>>>>>>> users can downgrade the flink version if meet problems using
>>> the
>>> > new
>>> > >>>>>>>> version. Of course, we will try our best to make the new
>>> release
>>> > out
>>> > >>>>>>>> of question.
>>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will
>>> make
>>> > our
>>> > >>>>>>>> build unlikely convergence as more and more build options
>>> will be
>>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
>>> > >>>>>>>>
>>> > >>>>>>>> What do you think?
>>> > >>>>>>>>
>>> > >>>>>>>> Best, Hequn
>>> > >>>>>>>>
>>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>>> > >> [hidden email]
>>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>>> > >>>>>>>>
>>> > >>>>>>>>        So here's the state of things:
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1,
>>> which
>>> > >>>>>>>>        eliminates a whole category of security
>>> vulnerabilities.
>>> > >>>>>>>>        The flink master works perfectly fine with that
>>> version;
>>> > 1.9
>>> > >> will
>>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>>> adjustment.
>>> > >>>>>>>>
>>> > >>>>>>>>        Hence, there may be value in first doing a flink-shaded
>>> > >>>>>>>> release so
>>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and
>>> 1.9.2 .
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>>        As for other jackson dependencies (coming from calcite,
>>> > kafka,
>>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of master
>>> > >> yesterday
>>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and they
>>> > >> passed. I
>>> > >>>>>>>>        will open a PR soon-ish for making this change on
>>> master.
>>> > >>>>>>>>
>>> > >>>>>>>>        The question now is whether we want to backport this
>>> > change to
>>> > >>>>>>>>        1.8/1.9 .
>>> > >>>>>>>>        Some code paths /may /not be covered by our tests, and
>>> > >> transitive
>>> > >>>>>>>>        jackson users /might /run into issues.
>>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>>> upgrade,
>>> > by
>>> > >>>>>>>>        adding a separate profile that bumps the versions. This
>>> > would
>>> > >>>>>>>>        present users/providers who are concerned about the
>>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of
>>> /some
>>> > >> /things
>>> > >>>>>>>>        /maybe /not working.
>>> > >>>>>>>>
>>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>>> > >>>>>>>>>        Hi Chesnay, Jincheng
>>> > >>>>>>>>>
>>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>>> > >>>>>>>>>        Thanks a lot for providing the information about the
>>> > security
>>> > >>>>>>>>>        vulnerabilities! @Chesnay
>>> > >>>>>>>>>
>>> > >>>>>>>>>        Best, Hequn
>>> > >>>>>>>>>
>>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>>> > >>>>>> [hidden email]> <mailto:[hidden email]>
>>> > >>>>>>>>>        wrote:
>>> > >>>>>>>>>
>>> > >>>>>>>>>>        +1 for try to eliminate the security vulnerabilities.
>>> > Great
>>> > >>>>>> thanks for
>>> > >>>>>>>>>>        doing this important work, Chesnay!
>>> > >>>>>>>>>>        What do you think Hequn ?
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>        Best,
>>> > >>>>>>>>>>        Jincheng
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>>> > >>>>>>>>>> <mailto:[hidden email]>
>>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>>> > >>>>>>>>>>>        It would be great if you could give me a day or 2 to
>>> > check
>>> > >> how
>>> > >>>>>> easy it
>>> > >>>>>>>>>>>        would be to bump the various jackson dependencies to
>>> > >>>>>>>>>>> eliminate a
>>> > >>>>>> few
>>> > >>>>>>>>>>>        security vulnerabilities.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>>> > >>>>>>>>>>>>        Hi Flink devs,
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
>>> > released.
>>> > >> So,
>>> > >>>>>> What do
>>> > >>>>>>>>>>        you
>>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        We already have many important bug fixes in the
>>> > >> release-1.8
>>> > >>>>>> branch (29
>>> > >>>>>>>>>>>>        resolved issues).
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        Most notable fixes are:
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't give
>>> up
>>> > >>>>>>>>>>>> leadership
>>> > >>>>>> when AM
>>> > >>>>>>>>>>>        is
>>> > >>>>>>>>>>>>        shut down
>>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>>> JobMaster.disconnectTaskManager
>>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo should
>>> > >> consider
>>> > >>>>>>>>>>        fieldsNames
>>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too
>>> Many
>>> > >>>>>>>>>>>> Containers
>>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the same
>>> > >>>>>> AllocationID leads
>>> > >>>>>>>>>>        to
>>> > >>>>>>>>>>>>        inconsistent slot table
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        Furthermore, the following critical issues is in
>>> > progress,
>>> > >>>>>> maybe we can
>>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
>>> > >>>>>> YarnResourceManager's
>>> > >>>>>>>>>>>        main
>>> > >>>>>>>>>>>>        thread
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        Please let me know what you think?
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>        Best,
>>> > >>>>>>>>>>>>        Jincheng
>>> > >>>>>>>>>>>>
>>> > >>
>>> >
>>> >
>>>
>>

[Ufuk Celebi-2]

@Hequn: flink-shaded:9.0 is available in Maven central now. I think you can
go ahead and create the first RC. :-)

On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <[hidden email]> wrote:

> Hi Hequn,
>
> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
> Given that this fix is for batch job only and batch is not very good in
> 1.8, I think it is a not blocker of release 1.8.3.
> So just don't be blocked by it and feel free to cut the RC when other
> blocking issues are resolved.
>
> Thanks,
> Zhu Zhu
>
> Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：
>
> > Hi Zhu Zhu,
> >
> > Thanks a lot for letting us know!
> > We can't cut the first RC right now due to the wait of the flink-shade
> > release, so go ahead.
> >
> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once the
> > release of flink-shade is done,
> > but I will try my best to have it in 1.8.3. Hope we can get it on board
> on
> > time. :)
> >
> > Best, Hequn
> >
> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
> >
> >> Hi Jincheng & Hequn
> >>
> >> Thanks for driving the releasing of 1.8.3.
> >>
> >> I am now working on FLINK-14735. The fix avoids duplicated input
> >> checking when scheduling ALL-to-ALL
> >> connected downstream consumers with ALL input constraints. The
> duplicated
> >> checking can cause severe
> >> performance issues for large scale jobs. So I hope the fix could be
> >> released with 1.8.3.
> >>
> >> The fix is already merged into master, and is now in the process of
> >> backporting to 1.8.
> >>
> >> Thanks,
> >> Zhu Zhu
> >>
> >> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
> >>
> >>> Thanks Chesnay.
> >>>
> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
> >>> version bump and leave those for a future release. Realistically, the
> >>> flink-shaded release will take until mid next week or end of next week.
> >>> But
> >>> please correct me if you think that it should not take that long or
> it's
> >>> OK
> >>> to block the 1.8.3 release on the flink-shaded release.
> >>>
> >>> – Ufuk
> >>>
> >>>
> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]>
> >>> wrote:
> >>>
> >>> > I've kicked off a discussion about the next flink-shaded release, and
> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
> >>> >
> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
> >>> > > That's great, thank you very much! Ideally, we can kick off the
> >>> release
> >>> > > vote for the first RC of 1.8.3 within next week. :)
> >>> > >
> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
> [hidden email]
> >>> >
> >>> > wrote:
> >>> > >
> >>> > >> I'm not aware of any more planned changes to flink-shaded; so we
> >>> could
> >>> > >> start the release right away.
> >>> > >>
> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
> >>> > >>> Hi,
> >>> > >>>
> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
> >>> approach
> >>> > for
> >>> > >>> 1.8/1.9.
> >>> > >>> @Ufuk Thank you for the nice summary.
> >>> > >>>
> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
> >>> first
> >>> > >> do a
> >>> > >>> flink-shaded release.
> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded with
> >>> > >> upgraded
> >>> > >>> Jackson?
> >>> > >>>
> >>> > >>> Best, Hequn
> >>> > >>>
> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
> >>> [hidden email]>
> >>> > >> wrote:
> >>> > >>>> One small modification: the flink-shaded upgrade does not have
> to
> >>> be
> >>> > >>>> part of the profile; since it is only intended for internal use
> >>> anyway
> >>> > >>>> (and thus has limited exposure) we can be pretty sure this
> doesn't
> >>> > break
> >>> > >>>> anything.
> >>> > >>>>
> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
> >>> > >>>>> Ufuk's summary is correct.
> >>> > >>>>>
> >>> > >>>>> There's a slight caveat in that we'd also have to bump the
> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
> >>> > >>>>> but I have no concerns about this change.
> >>> > >>>>>
> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the
> >>> > >>>>>> profiles in
> >>> > >>>>>> 1.8 and 1.9 without changing the default versions (including
> the
> >>> > >> default
> >>> > >>>>>> version of flink-shaded).
> >>> > >>>>>>
> >>> > >>>>>> As far as I can tell, the next steps would be:
> >>> > >>>>>>
> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master
> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
> profiles
> >>> > >>>>>> should also
> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
> >>> > >>>>>>
> >>> > >>>>>> @Chesnay: is this a correct summary?
> >>> > >>>>>>
> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
> >>> upside, we
> >>> > >>>>>> might
> >>> > >>>>>> get some additional feedback until the 1.10 release with these
> >>> > >>>>>> profiles in
> >>> > >>>>>> case users make use of them with 1.8/1.9.
> >>> > >>>>>>
> >>> > >>>>>> – Ufuk
> >>> > >>>>>>
> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
> >>> > [hidden email]
> >>> > >>>>>> wrote:
> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on
> >>> master
> >>> > >>>>>>> (and
> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
> >>> > >>>>>>>
> >>> > >>>>>>> I have only proposed it as an opt-in because a) we usually do
> >>> not
> >>> > >> bump
> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term
> change
> >>> > that
> >>> > >> we
> >>> > >>>>>>> aren't allowing to mature properly.
> >>> > >>>>>>> In contrast, the 1.10 release is significantly further away,
> >>> hence
> >>> > no
> >>> > >>>>>>> opt-in.
> >>> > >>>>>>>
> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being
> more
> >>> > >> common
> >>> > >>>>>>> in the future.
> >>> > >>>>>>>
> >>> > >>>>>>> We can certainly support every jackson version that fixes
> these
> >>> > >>>>>>> vulnerabilities; individual modules can always use a
> different
> >>> > >> version
> >>> > >>>>>>> (that hopefully includes the fixes).
> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may
> >>> or may
> >>> > >> not
> >>> > >>>>>>> be feasible.
> >>> > >>>>>>>
> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
> >>> > >>>>>>>> Hi Chesnay,
> >>> > >>>>>>>>
> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
> >>> Really a
> >>> > >> good
> >>> > >>>>>> job!
> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9
> >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities,
> >>> some of
> >>> > >> them
> >>> > >>>>>>>> seem can lead to high-security problems, thus from my point
> of
> >>> > view,
> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I
> would
> >>> like
> >>> > >> to
> >>> > >>>>>>>> trust your judgment as you are more professional at this
> >>> problem.
> >>> > >>>>>>>>
> >>> > >>>>>>>> - How to port this change to 1.8/1.9
> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another
> >>> > question
> >>> > >>>>>>>> here is whether do we plan to support multi jackson versions
> >>> that
> >>> > >> have
> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to
> >>> > support
> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an
> >>> > option,
> >>> > >>>>>>>> users can downgrade the flink version if meet problems using
> >>> the
> >>> > new
> >>> > >>>>>>>> version. Of course, we will try our best to make the new
> >>> release
> >>> > out
> >>> > >>>>>>>> of question.
> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will
> >>> make
> >>> > our
> >>> > >>>>>>>> build unlikely convergence as more and more build options
> >>> will be
> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
> >>> > >>>>>>>>
> >>> > >>>>>>>> What do you think?
> >>> > >>>>>>>>
> >>> > >>>>>>>> Best, Hequn
> >>> > >>>>>>>>
> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
> >>> > >> [hidden email]
> >>> > >>>>>>>> <mailto:[hidden email]>> wrote:
> >>> > >>>>>>>>
> >>> > >>>>>>>>        So here's the state of things:
> >>> > >>>>>>>>
> >>> > >>>>>>>>
> >>> > >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1,
> >>> which
> >>> > >>>>>>>>        eliminates a whole category of security
> >>> vulnerabilities.
> >>> > >>>>>>>>        The flink master works perfectly fine with that
> >>> version;
> >>> > 1.9
> >>> > >> will
> >>> > >>>>>>>>        likely do so too and 1.8 would require a minor
> >>> adjustment.
> >>> > >>>>>>>>
> >>> > >>>>>>>>        Hence, there may be value in first doing a
> flink-shaded
> >>> > >>>>>>>> release so
> >>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and
> >>> 1.9.2 .
> >>> > >>>>>>>>
> >>> > >>>>>>>>
> >>> > >>>>>>>>        As for other jackson dependencies (coming from
> calcite,
> >>> > kafka,
> >>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of
> master
> >>> > >> yesterday
> >>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and
> they
> >>> > >> passed. I
> >>> > >>>>>>>>        will open a PR soon-ish for making this change on
> >>> master.
> >>> > >>>>>>>>
> >>> > >>>>>>>>        The question now is whether we want to backport this
> >>> > change to
> >>> > >>>>>>>>        1.8/1.9 .
> >>> > >>>>>>>>        Some code paths /may /not be covered by our tests,
> and
> >>> > >> transitive
> >>> > >>>>>>>>        jackson users /might /run into issues.
> >>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
> >>> upgrade,
> >>> > by
> >>> > >>>>>>>>        adding a separate profile that bumps the versions.
> This
> >>> > would
> >>> > >>>>>>>>        present users/providers who are concerned about the
> >>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of
> >>> /some
> >>> > >> /things
> >>> > >>>>>>>>        /maybe /not working.
> >>> > >>>>>>>>
> >>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
> >>> > >>>>>>>>>        Hi Chesnay, Jincheng
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
> >>> > >>>>>>>>>        Thanks a lot for providing the information about the
> >>> > security
> >>> > >>>>>>>>>        vulnerabilities! @Chesnay
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>        Best, Hequn
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
> >>> > >>>>>> [hidden email]> <mailto:[hidden email]>
> >>> > >>>>>>>>>        wrote:
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>>        +1 for try to eliminate the security
> vulnerabilities.
> >>> > Great
> >>> > >>>>>> thanks for
> >>> > >>>>>>>>>>        doing this important work, Chesnay!
> >>> > >>>>>>>>>>        What do you think Hequn ?
> >>> > >>>>>>>>>>
> >>> > >>>>>>>>>>        Best,
> >>> > >>>>>>>>>>        Jincheng
> >>> > >>>>>>>>>>
> >>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
> >>> > >>>>>>>>>> <mailto:[hidden email]>
> >>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
> >>> > >>>>>>>>>>>        It would be great if you could give me a day or 2
> to
> >>> > check
> >>> > >> how
> >>> > >>>>>> easy it
> >>> > >>>>>>>>>>>        would be to bump the various jackson dependencies
> to
> >>> > >>>>>>>>>>> eliminate a
> >>> > >>>>>> few
> >>> > >>>>>>>>>>>        security vulnerabilities.
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
> >>> > >>>>>>>>>>>>        Hi Flink devs,
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
> >>> > released.
> >>> > >> So,
> >>> > >>>>>> What do
> >>> > >>>>>>>>>>        you
> >>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        We already have many important bug fixes in the
> >>> > >> release-1.8
> >>> > >>>>>> branch (29
> >>> > >>>>>>>>>>>>        resolved issues).
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        Most notable fixes are:
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't give
> >>> up
> >>> > >>>>>>>>>>>> leadership
> >>> > >>>>>> when AM
> >>> > >>>>>>>>>>>        is
> >>> > >>>>>>>>>>>>        shut down
> >>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
> >>> JobMaster.disconnectTaskManager
> >>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo
> should
> >>> > >> consider
> >>> > >>>>>>>>>>        fieldsNames
> >>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too
> >>> Many
> >>> > >>>>>>>>>>>> Containers
> >>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the
> same
> >>> > >>>>>> AllocationID leads
> >>> > >>>>>>>>>>        to
> >>> > >>>>>>>>>>>>        inconsistent slot table
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        Furthermore, the following critical issues is in
> >>> > progress,
> >>> > >>>>>> maybe we can
> >>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
> >>> > >>>>>> YarnResourceManager's
> >>> > >>>>>>>>>>>        main
> >>> > >>>>>>>>>>>>        thread
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        Please let me know what you think?
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>        Best,
> >>> > >>>>>>>>>>>>        Jincheng
> >>> > >>>>>>>>>>>>
> >>> > >>
> >>> >
> >>> >
> >>>
> >>
>

[Hequn Cheng]

Hi,

@Ufuk Celebi <[hidden email]> Hi, we are very close now. There is one issue(
FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>) left that
I want to double-check with you guys. Once this is done, we can create the
first RC. I already have some minor comments in the PR
<https://github.com/apache/flink/pull/10195>.

@Zhu Zhu <[hidden email]> Glad to hear that it is not a blocker. Thank
you.

Best, Hequn

On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <[hidden email]> wrote:

> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you
> can go ahead and create the first RC. :-)
>
> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <[hidden email]> wrote:
>
>> Hi Hequn,
>>
>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
>> Given that this fix is for batch job only and batch is not very good in
>> 1.8, I think it is a not blocker of release 1.8.3.
>> So just don't be blocked by it and feel free to cut the RC when other
>> blocking issues are resolved.
>>
>> Thanks,
>> Zhu Zhu
>>
>> Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：
>>
>> > Hi Zhu Zhu,
>> >
>> > Thanks a lot for letting us know!
>> > We can't cut the first RC right now due to the wait of the flink-shade
>> > release, so go ahead.
>> >
>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once
>> the
>> > release of flink-shade is done,
>> > but I will try my best to have it in 1.8.3. Hope we can get it on board
>> on
>> > time. :)
>> >
>> > Best, Hequn
>> >
>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
>> >
>> >> Hi Jincheng & Hequn
>> >>
>> >> Thanks for driving the releasing of 1.8.3.
>> >>
>> >> I am now working on FLINK-14735. The fix avoids duplicated input
>> >> checking when scheduling ALL-to-ALL
>> >> connected downstream consumers with ALL input constraints. The
>> duplicated
>> >> checking can cause severe
>> >> performance issues for large scale jobs. So I hope the fix could be
>> >> released with 1.8.3.
>> >>
>> >> The fix is already merged into master, and is now in the process of
>> >> backporting to 1.8.
>> >>
>> >> Thanks,
>> >> Zhu Zhu
>> >>
>> >> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>> >>
>> >>> Thanks Chesnay.
>> >>>
>> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
>> >>> version bump and leave those for a future release. Realistically, the
>> >>> flink-shaded release will take until mid next week or end of next
>> week.
>> >>> But
>> >>> please correct me if you think that it should not take that long or
>> it's
>> >>> OK
>> >>> to block the 1.8.3 release on the flink-shaded release.
>> >>>
>> >>> – Ufuk
>> >>>
>> >>>
>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]>
>> >>> wrote:
>> >>>
>> >>> > I've kicked off a discussion about the next flink-shaded release,
>> and
>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>> >>> >
>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>> >>> > > That's great, thank you very much! Ideally, we can kick off the
>> >>> release
>> >>> > > vote for the first RC of 1.8.3 within next week. :)
>> >>> > >
>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
>> [hidden email]
>> >>> >
>> >>> > wrote:
>> >>> > >
>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so we
>> >>> could
>> >>> > >> start the release right away.
>> >>> > >>
>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>> >>> > >>> Hi,
>> >>> > >>>
>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>> >>> approach
>> >>> > for
>> >>> > >>> 1.8/1.9.
>> >>> > >>> @Ufuk Thank you for the nice summary.
>> >>> > >>>
>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
>> >>> first
>> >>> > >> do a
>> >>> > >>> flink-shaded release.
>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded
>> with
>> >>> > >> upgraded
>> >>> > >>> Jackson?
>> >>> > >>>
>> >>> > >>> Best, Hequn
>> >>> > >>>
>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>> >>> [hidden email]>
>> >>> > >> wrote:
>> >>> > >>>> One small modification: the flink-shaded upgrade does not have
>> to
>> >>> be
>> >>> > >>>> part of the profile; since it is only intended for internal use
>> >>> anyway
>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this
>> doesn't
>> >>> > break
>> >>> > >>>> anything.
>> >>> > >>>>
>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>> >>> > >>>>> Ufuk's summary is correct.
>> >>> > >>>>>
>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the
>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>> >>> > >>>>> but I have no concerns about this change.
>> >>> > >>>>>
>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the
>> >>> > >>>>>> profiles in
>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions (including
>> the
>> >>> > >> default
>> >>> > >>>>>> version of flink-shaded).
>> >>> > >>>>>>
>> >>> > >>>>>> As far as I can tell, the next steps would be:
>> >>> > >>>>>>
>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
>> profiles
>> >>> > >>>>>> should also
>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
>> >>> > >>>>>>
>> >>> > >>>>>> @Chesnay: is this a correct summary?
>> >>> > >>>>>>
>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
>> >>> upside, we
>> >>> > >>>>>> might
>> >>> > >>>>>> get some additional feedback until the 1.10 release with
>> these
>> >>> > >>>>>> profiles in
>> >>> > >>>>>> case users make use of them with 1.8/1.9.
>> >>> > >>>>>>
>> >>> > >>>>>> – Ufuk
>> >>> > >>>>>>
>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>> >>> > [hidden email]
>> >>> > >>>>>> wrote:
>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on
>> >>> master
>> >>> > >>>>>>> (and
>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>> >>> > >>>>>>>
>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we usually
>> do
>> >>> not
>> >>> > >> bump
>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term
>> change
>> >>> > that
>> >>> > >> we
>> >>> > >>>>>>> aren't allowing to mature properly.
>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further away,
>> >>> hence
>> >>> > no
>> >>> > >>>>>>> opt-in.
>> >>> > >>>>>>>
>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being
>> more
>> >>> > >> common
>> >>> > >>>>>>> in the future.
>> >>> > >>>>>>>
>> >>> > >>>>>>> We can certainly support every jackson version that fixes
>> these
>> >>> > >>>>>>> vulnerabilities; individual modules can always use a
>> different
>> >>> > >> version
>> >>> > >>>>>>> (that hopefully includes the fixes).
>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may
>> >>> or may
>> >>> > >> not
>> >>> > >>>>>>> be feasible.
>> >>> > >>>>>>>
>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>> >>> > >>>>>>>> Hi Chesnay,
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
>> >>> Really a
>> >>> > >> good
>> >>> > >>>>>> job!
>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>> >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities,
>> >>> some of
>> >>> > >> them
>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my
>> point of
>> >>> > view,
>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I
>> would
>> >>> like
>> >>> > >> to
>> >>> > >>>>>>>> trust your judgment as you are more professional at this
>> >>> problem.
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> - How to port this change to 1.8/1.9
>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another
>> >>> > question
>> >>> > >>>>>>>> here is whether do we plan to support multi jackson
>> versions
>> >>> that
>> >>> > >> have
>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to
>> >>> > support
>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an
>> >>> > option,
>> >>> > >>>>>>>> users can downgrade the flink version if meet problems
>> using
>> >>> the
>> >>> > new
>> >>> > >>>>>>>> version. Of course, we will try our best to make the new
>> >>> release
>> >>> > out
>> >>> > >>>>>>>> of question.
>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will
>> >>> make
>> >>> > our
>> >>> > >>>>>>>> build unlikely convergence as more and more build options
>> >>> will be
>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> What do you think?
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> Best, Hequn
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>> >>> > >> [hidden email]
>> >>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>        So here's the state of things:
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1,
>> >>> which
>> >>> > >>>>>>>>        eliminates a whole category of security
>> >>> vulnerabilities.
>> >>> > >>>>>>>>        The flink master works perfectly fine with that
>> >>> version;
>> >>> > 1.9
>> >>> > >> will
>> >>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>> >>> adjustment.
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>        Hence, there may be value in first doing a
>> flink-shaded
>> >>> > >>>>>>>> release so
>> >>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and
>> >>> 1.9.2 .
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>        As for other jackson dependencies (coming from
>> calcite,
>> >>> > kafka,
>> >>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of
>> master
>> >>> > >> yesterday
>> >>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and
>> they
>> >>> > >> passed. I
>> >>> > >>>>>>>>        will open a PR soon-ish for making this change on
>> >>> master.
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>        The question now is whether we want to backport this
>> >>> > change to
>> >>> > >>>>>>>>        1.8/1.9 .
>> >>> > >>>>>>>>        Some code paths /may /not be covered by our tests,
>> and
>> >>> > >> transitive
>> >>> > >>>>>>>>        jackson users /might /run into issues.
>> >>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>> >>> upgrade,
>> >>> > by
>> >>> > >>>>>>>>        adding a separate profile that bumps the versions.
>> This
>> >>> > would
>> >>> > >>>>>>>>        present users/providers who are concerned about the
>> >>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of
>> >>> /some
>> >>> > >> /things
>> >>> > >>>>>>>>        /maybe /not working.
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>> >>> > >>>>>>>>>        Hi Chesnay, Jincheng
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>> >>> > >>>>>>>>>        Thanks a lot for providing the information about
>> the
>> >>> > security
>> >>> > >>>>>>>>>        vulnerabilities! @Chesnay
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>        Best, Hequn
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>> >>> > >>>>>> [hidden email]> <mailto:[hidden email]>
>> >>> > >>>>>>>>>        wrote:
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>>        +1 for try to eliminate the security
>> vulnerabilities.
>> >>> > Great
>> >>> > >>>>>> thanks for
>> >>> > >>>>>>>>>>        doing this important work, Chesnay!
>> >>> > >>>>>>>>>>        What do you think Hequn ?
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>        Best,
>> >>> > >>>>>>>>>>        Jincheng
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>> >>> > >>>>>>>>>> <mailto:[hidden email]>
>> >>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>> >>> > >>>>>>>>>>>        It would be great if you could give me a day or
>> 2 to
>> >>> > check
>> >>> > >> how
>> >>> > >>>>>> easy it
>> >>> > >>>>>>>>>>>        would be to bump the various jackson
>> dependencies to
>> >>> > >>>>>>>>>>> eliminate a
>> >>> > >>>>>> few
>> >>> > >>>>>>>>>>>        security vulnerabilities.
>> >>> > >>>>>>>>>>>
>> >>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>> >>> > >>>>>>>>>>>>        Hi Flink devs,
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
>> >>> > released.
>> >>> > >> So,
>> >>> > >>>>>> What do
>> >>> > >>>>>>>>>>        you
>> >>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        We already have many important bug fixes in the
>> >>> > >> release-1.8
>> >>> > >>>>>> branch (29
>> >>> > >>>>>>>>>>>>        resolved issues).
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        Most notable fixes are:
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't
>> give
>> >>> up
>> >>> > >>>>>>>>>>>> leadership
>> >>> > >>>>>> when AM
>> >>> > >>>>>>>>>>>        is
>> >>> > >>>>>>>>>>>>        shut down
>> >>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>> >>> JobMaster.disconnectTaskManager
>> >>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo
>> should
>> >>> > >> consider
>> >>> > >>>>>>>>>>        fieldsNames
>> >>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires Too
>> >>> Many
>> >>> > >>>>>>>>>>>> Containers
>> >>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the
>> same
>> >>> > >>>>>> AllocationID leads
>> >>> > >>>>>>>>>>        to
>> >>> > >>>>>>>>>>>>        inconsistent slot table
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        Furthermore, the following critical issues is in
>> >>> > progress,
>> >>> > >>>>>> maybe we can
>> >>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks the
>> >>> > >>>>>> YarnResourceManager's
>> >>> > >>>>>>>>>>>        main
>> >>> > >>>>>>>>>>>>        thread
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        Please let me know what you think?
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>        Best,
>> >>> > >>>>>>>>>>>>        Jincheng
>> >>> > >>>>>>>>>>>>
>> >>> > >>
>> >>> >
>> >>> >
>> >>>
>> >>
>>
>

[Hequn Cheng]

Hi all,

I would like to share with you that all blockers are resolved now. If there
are no more critical issues, I will create the first RC tomorrow and vote
on it directly.
Hope everything goes well!

Thank you all for the help of fixing, reviewing, driving and discussions!

Best, Hequn

On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <[hidden email]> wrote:

> Hi,
>
> @Ufuk Celebi <[hidden email]> Hi, we are very close now. There is one
> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>)
> left that I want to double-check with you guys. Once this is done, we can
> create the first RC. I already have some minor comments in the PR
> <https://github.com/apache/flink/pull/10195>.
>
> @Zhu Zhu <[hidden email]> Glad to hear that it is not a blocker. Thank
> you.
>
> Best, Hequn
>
> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <[hidden email]> wrote:
>
>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you
>> can go ahead and create the first RC. :-)
>>
>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <[hidden email]> wrote:
>>
>>> Hi Hequn,
>>>
>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
>>> Given that this fix is for batch job only and batch is not very good in
>>> 1.8, I think it is a not blocker of release 1.8.3.
>>> So just don't be blocked by it and feel free to cut the RC when other
>>> blocking issues are resolved.
>>>
>>> Thanks,
>>> Zhu Zhu
>>>
>>> Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：
>>>
>>> > Hi Zhu Zhu,
>>> >
>>> > Thanks a lot for letting us know!
>>> > We can't cut the first RC right now due to the wait of the flink-shade
>>> > release, so go ahead.
>>> >
>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once
>>> the
>>> > release of flink-shade is done,
>>> > but I will try my best to have it in 1.8.3. Hope we can get it on
>>> board on
>>> > time. :)
>>> >
>>> > Best, Hequn
>>> >
>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
>>> >
>>> >> Hi Jincheng & Hequn
>>> >>
>>> >> Thanks for driving the releasing of 1.8.3.
>>> >>
>>> >> I am now working on FLINK-14735. The fix avoids duplicated input
>>> >> checking when scheduling ALL-to-ALL
>>> >> connected downstream consumers with ALL input constraints. The
>>> duplicated
>>> >> checking can cause severe
>>> >> performance issues for large scale jobs. So I hope the fix could be
>>> >> released with 1.8.3.
>>> >>
>>> >> The fix is already merged into master, and is now in the process of
>>> >> backporting to 1.8.
>>> >>
>>> >> Thanks,
>>> >> Zhu Zhu
>>> >>
>>> >> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>>> >>
>>> >>> Thanks Chesnay.
>>> >>>
>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
>>> >>> version bump and leave those for a future release. Realistically, the
>>> >>> flink-shaded release will take until mid next week or end of next
>>> week.
>>> >>> But
>>> >>> please correct me if you think that it should not take that long or
>>> it's
>>> >>> OK
>>> >>> to block the 1.8.3 release on the flink-shaded release.
>>> >>>
>>> >>> – Ufuk
>>> >>>
>>> >>>
>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <[hidden email]
>>> >
>>> >>> wrote:
>>> >>>
>>> >>> > I've kicked off a discussion about the next flink-shaded release,
>>> and
>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>>> >>> >
>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>>> >>> > > That's great, thank you very much! Ideally, we can kick off the
>>> >>> release
>>> >>> > > vote for the first RC of 1.8.3 within next week. :)
>>> >>> > >
>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
>>> [hidden email]
>>> >>> >
>>> >>> > wrote:
>>> >>> > >
>>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so we
>>> >>> could
>>> >>> > >> start the release right away.
>>> >>> > >>
>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>>> >>> > >>> Hi,
>>> >>> > >>>
>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>>> >>> approach
>>> >>> > for
>>> >>> > >>> 1.8/1.9.
>>> >>> > >>> @Ufuk Thank you for the nice summary.
>>> >>> > >>>
>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit
>>> to
>>> >>> first
>>> >>> > >> do a
>>> >>> > >>> flink-shaded release.
>>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded
>>> with
>>> >>> > >> upgraded
>>> >>> > >>> Jackson?
>>> >>> > >>>
>>> >>> > >>> Best, Hequn
>>> >>> > >>>
>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>>> >>> [hidden email]>
>>> >>> > >> wrote:
>>> >>> > >>>> One small modification: the flink-shaded upgrade does not
>>> have to
>>> >>> be
>>> >>> > >>>> part of the profile; since it is only intended for internal
>>> use
>>> >>> anyway
>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this
>>> doesn't
>>> >>> > break
>>> >>> > >>>> anything.
>>> >>> > >>>>
>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>>> >>> > >>>>> Ufuk's summary is correct.
>>> >>> > >>>>>
>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the
>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>>> >>> > >>>>> but I have no concerns about this change.
>>> >>> > >>>>>
>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include
>>> the
>>> >>> > >>>>>> profiles in
>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions
>>> (including the
>>> >>> > >> default
>>> >>> > >>>>>> version of flink-shaded).
>>> >>> > >>>>>>
>>> >>> > >>>>>> As far as I can tell, the next steps would be:
>>> >>> > >>>>>>
>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
>>> profiles
>>> >>> > >>>>>> should also
>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
>>> >>> > >>>>>>
>>> >>> > >>>>>> @Chesnay: is this a correct summary?
>>> >>> > >>>>>>
>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
>>> >>> upside, we
>>> >>> > >>>>>> might
>>> >>> > >>>>>> get some additional feedback until the 1.10 release with
>>> these
>>> >>> > >>>>>> profiles in
>>> >>> > >>>>>> case users make use of them with 1.8/1.9.
>>> >>> > >>>>>>
>>> >>> > >>>>>> – Ufuk
>>> >>> > >>>>>>
>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>>> >>> > [hidden email]
>>> >>> > >>>>>> wrote:
>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2;
>>> on
>>> >>> master
>>> >>> > >>>>>>> (and
>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>>> >>> > >>>>>>>
>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we usually
>>> do
>>> >>> not
>>> >>> > >> bump
>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term
>>> change
>>> >>> > that
>>> >>> > >> we
>>> >>> > >>>>>>> aren't allowing to mature properly.
>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further
>>> away,
>>> >>> hence
>>> >>> > no
>>> >>> > >>>>>>> opt-in.
>>> >>> > >>>>>>>
>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being
>>> more
>>> >>> > >> common
>>> >>> > >>>>>>> in the future.
>>> >>> > >>>>>>>
>>> >>> > >>>>>>> We can certainly support every jackson version that fixes
>>> these
>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a
>>> different
>>> >>> > >> version
>>> >>> > >>>>>>> (that hopefully includes the fixes).
>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that
>>> may
>>> >>> or may
>>> >>> > >> not
>>> >>> > >>>>>>> be feasible.
>>> >>> > >>>>>>>
>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>>> >>> > >>>>>>>> Hi Chesnay,
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
>>> >>> Really a
>>> >>> > >> good
>>> >>> > >>>>>> job!
>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>>> >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities,
>>> >>> some of
>>> >>> > >> them
>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my
>>> point of
>>> >>> > view,
>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I
>>> would
>>> >>> like
>>> >>> > >> to
>>> >>> > >>>>>>>> trust your judgment as you are more professional at this
>>> >>> problem.
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9
>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea.
>>> Another
>>> >>> > question
>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson
>>> versions
>>> >>> that
>>> >>> > >> have
>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan
>>> to
>>> >>> > support
>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As
>>> an
>>> >>> > option,
>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems
>>> using
>>> >>> the
>>> >>> > new
>>> >>> > >>>>>>>> version. Of course, we will try our best to make the new
>>> >>> release
>>> >>> > out
>>> >>> > >>>>>>>> of question.
>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will
>>> >>> make
>>> >>> > our
>>> >>> > >>>>>>>> build unlikely convergence as more and more build options
>>> >>> will be
>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>> What do you think?
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>> Best, Hequn
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>>> >>> > >> [hidden email]
>>> >>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>        So here's the state of things:
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>        The master of flink-shaded now uses jackson 2.10.1,
>>> >>> which
>>> >>> > >>>>>>>>        eliminates a whole category of security
>>> >>> vulnerabilities.
>>> >>> > >>>>>>>>        The flink master works perfectly fine with that
>>> >>> version;
>>> >>> > 1.9
>>> >>> > >> will
>>> >>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>>> >>> adjustment.
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>        Hence, there may be value in first doing a
>>> flink-shaded
>>> >>> > >>>>>>>> release so
>>> >>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3 and
>>> >>> 1.9.2 .
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>        As for other jackson dependencies (coming from
>>> calcite,
>>> >>> > kafka,
>>> >>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of
>>> master
>>> >>> > >> yesterday
>>> >>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1, and
>>> they
>>> >>> > >> passed. I
>>> >>> > >>>>>>>>        will open a PR soon-ish for making this change on
>>> >>> master.
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>        The question now is whether we want to backport
>>> this
>>> >>> > change to
>>> >>> > >>>>>>>>        1.8/1.9 .
>>> >>> > >>>>>>>>        Some code paths /may /not be covered by our tests,
>>> and
>>> >>> > >> transitive
>>> >>> > >>>>>>>>        jackson users /might /run into issues.
>>> >>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>>> >>> upgrade,
>>> >>> > by
>>> >>> > >>>>>>>>        adding a separate profile that bumps the versions.
>>> This
>>> >>> > would
>>> >>> > >>>>>>>>        present users/providers who are concerned about the
>>> >>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of
>>> >>> /some
>>> >>> > >> /things
>>> >>> > >>>>>>>>        /maybe /not working.
>>> >>> > >>>>>>>>
>>> >>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>>> >>> > >>>>>>>>>        Hi Chesnay, Jincheng
>>> >>> > >>>>>>>>>
>>> >>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>>> >>> > >>>>>>>>>        Thanks a lot for providing the information about
>>> the
>>> >>> > security
>>> >>> > >>>>>>>>>        vulnerabilities! @Chesnay
>>> >>> > >>>>>>>>>
>>> >>> > >>>>>>>>>        Best, Hequn
>>> >>> > >>>>>>>>>
>>> >>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>>> >>> > >>>>>> [hidden email]> <mailto:[hidden email]>
>>> >>> > >>>>>>>>>        wrote:
>>> >>> > >>>>>>>>>
>>> >>> > >>>>>>>>>>        +1 for try to eliminate the security
>>> vulnerabilities.
>>> >>> > Great
>>> >>> > >>>>>> thanks for
>>> >>> > >>>>>>>>>>        doing this important work, Chesnay!
>>> >>> > >>>>>>>>>>        What do you think Hequn ?
>>> >>> > >>>>>>>>>>
>>> >>> > >>>>>>>>>>        Best,
>>> >>> > >>>>>>>>>>        Jincheng
>>> >>> > >>>>>>>>>>
>>> >>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>>> >>> > >>>>>>>>>> <mailto:[hidden email]>
>>> >>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>>> >>> > >>>>>>>>>>>        It would be great if you could give me a day or
>>> 2 to
>>> >>> > check
>>> >>> > >> how
>>> >>> > >>>>>> easy it
>>> >>> > >>>>>>>>>>>        would be to bump the various jackson
>>> dependencies to
>>> >>> > >>>>>>>>>>> eliminate a
>>> >>> > >>>>>> few
>>> >>> > >>>>>>>>>>>        security vulnerabilities.
>>> >>> > >>>>>>>>>>>
>>> >>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>>> >>> > >>>>>>>>>>>>        Hi Flink devs,
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
>>> >>> > released.
>>> >>> > >> So,
>>> >>> > >>>>>> What do
>>> >>> > >>>>>>>>>>        you
>>> >>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        We already have many important bug fixes in the
>>> >>> > >> release-1.8
>>> >>> > >>>>>> branch (29
>>> >>> > >>>>>>>>>>>>        resolved issues).
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        Most notable fixes are:
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't
>>> give
>>> >>> up
>>> >>> > >>>>>>>>>>>> leadership
>>> >>> > >>>>>> when AM
>>> >>> > >>>>>>>>>>>        is
>>> >>> > >>>>>>>>>>>>        shut down
>>> >>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>>> >>> JobMaster.disconnectTaskManager
>>> >>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo
>>> should
>>> >>> > >> consider
>>> >>> > >>>>>>>>>>        fieldsNames
>>> >>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires
>>> Too
>>> >>> Many
>>> >>> > >>>>>>>>>>>> Containers
>>> >>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with the
>>> same
>>> >>> > >>>>>> AllocationID leads
>>> >>> > >>>>>>>>>>        to
>>> >>> > >>>>>>>>>>>>        inconsistent slot table
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        Furthermore, the following critical issues is
>>> in
>>> >>> > progress,
>>> >>> > >>>>>> maybe we can
>>> >>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks
>>> the
>>> >>> > >>>>>> YarnResourceManager's
>>> >>> > >>>>>>>>>>>        main
>>> >>> > >>>>>>>>>>>>        thread
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        Please let me know what you think?
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>>>>>>>>>>>        Best,
>>> >>> > >>>>>>>>>>>>        Jincheng
>>> >>> > >>>>>>>>>>>>
>>> >>> > >>
>>> >>> >
>>> >>> >
>>> >>>
>>> >>
>>>
>>

[jincheng sun]

Hi Hequn,

Thank you for your great job! Looking forward the first RC of 1.8.3 !
BTW: The version of 1.8.4 already created here:
https://issues.apache.org/jira/projects/FLINK/versions/12346552

Best,
Jincheng

Hequn Cheng <[hidden email]> 于2019年11月26日周二 下午8:18写道：

> Hi all,
>
> I would like to share with you that all blockers are resolved now. If
> there are no more critical issues, I will create the first RC tomorrow and
> vote on it directly.
> Hope everything goes well!
>
> Thank you all for the help of fixing, reviewing, driving and discussions!
>
> Best, Hequn
>
> On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <[hidden email]> wrote:
>
>> Hi,
>>
>> @Ufuk Celebi <[hidden email]> Hi, we are very close now. There is one
>> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>)
>> left that I want to double-check with you guys. Once this is done, we can
>> create the first RC. I already have some minor comments in the PR
>> <https://github.com/apache/flink/pull/10195>.
>>
>> @Zhu Zhu <[hidden email]> Glad to hear that it is not a blocker.
>> Thank you.
>>
>> Best, Hequn
>>
>> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <[hidden email]> wrote:
>>
>>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you
>>> can go ahead and create the first RC. :-)
>>>
>>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <[hidden email]> wrote:
>>>
>>>> Hi Hequn,
>>>>
>>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
>>>> Given that this fix is for batch job only and batch is not very good in
>>>> 1.8, I think it is a not blocker of release 1.8.3.
>>>> So just don't be blocked by it and feel free to cut the RC when other
>>>> blocking issues are resolved.
>>>>
>>>> Thanks,
>>>> Zhu Zhu
>>>>
>>>> Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：
>>>>
>>>> > Hi Zhu Zhu,
>>>> >
>>>> > Thanks a lot for letting us know!
>>>> > We can't cut the first RC right now due to the wait of the flink-shade
>>>> > release, so go ahead.
>>>> >
>>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once
>>>> the
>>>> > release of flink-shade is done,
>>>> > but I will try my best to have it in 1.8.3. Hope we can get it on
>>>> board on
>>>> > time. :)
>>>> >
>>>> > Best, Hequn
>>>> >
>>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
>>>> >
>>>> >> Hi Jincheng & Hequn
>>>> >>
>>>> >> Thanks for driving the releasing of 1.8.3.
>>>> >>
>>>> >> I am now working on FLINK-14735. The fix avoids duplicated input
>>>> >> checking when scheduling ALL-to-ALL
>>>> >> connected downstream consumers with ALL input constraints. The
>>>> duplicated
>>>> >> checking can cause severe
>>>> >> performance issues for large scale jobs. So I hope the fix could be
>>>> >> released with 1.8.3.
>>>> >>
>>>> >> The fix is already merged into master, and is now in the process of
>>>> >> backporting to 1.8.
>>>> >>
>>>> >> Thanks,
>>>> >> Zhu Zhu
>>>> >>
>>>> >> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>>>> >>
>>>> >>> Thanks Chesnay.
>>>> >>>
>>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the
>>>> Jackson
>>>> >>> version bump and leave those for a future release. Realistically,
>>>> the
>>>> >>> flink-shaded release will take until mid next week or end of next
>>>> week.
>>>> >>> But
>>>> >>> please correct me if you think that it should not take that long or
>>>> it's
>>>> >>> OK
>>>> >>> to block the 1.8.3 release on the flink-shaded release.
>>>> >>>
>>>> >>> – Ufuk
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <
>>>> [hidden email]>
>>>> >>> wrote:
>>>> >>>
>>>> >>> > I've kicked off a discussion about the next flink-shaded release,
>>>> and
>>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>>>> >>> >
>>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>>>> >>> > > That's great, thank you very much! Ideally, we can kick off the
>>>> >>> release
>>>> >>> > > vote for the first RC of 1.8.3 within next week. :)
>>>> >>> > >
>>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
>>>> [hidden email]
>>>> >>> >
>>>> >>> > wrote:
>>>> >>> > >
>>>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so
>>>> we
>>>> >>> could
>>>> >>> > >> start the release right away.
>>>> >>> > >>
>>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>>>> >>> > >>> Hi,
>>>> >>> > >>>
>>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>>>> >>> approach
>>>> >>> > for
>>>> >>> > >>> 1.8/1.9.
>>>> >>> > >>> @Ufuk Thank you for the nice summary.
>>>> >>> > >>>
>>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit
>>>> to
>>>> >>> first
>>>> >>> > >> do a
>>>> >>> > >>> flink-shaded release.
>>>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded
>>>> with
>>>> >>> > >> upgraded
>>>> >>> > >>> Jackson?
>>>> >>> > >>>
>>>> >>> > >>> Best, Hequn
>>>> >>> > >>>
>>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>>>> >>> [hidden email]>
>>>> >>> > >> wrote:
>>>> >>> > >>>> One small modification: the flink-shaded upgrade does not
>>>> have to
>>>> >>> be
>>>> >>> > >>>> part of the profile; since it is only intended for internal
>>>> use
>>>> >>> anyway
>>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this
>>>> doesn't
>>>> >>> > break
>>>> >>> > >>>> anything.
>>>> >>> > >>>>
>>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>>>> >>> > >>>>> Ufuk's summary is correct.
>>>> >>> > >>>>>
>>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the
>>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>>>> >>> > >>>>> but I have no concerns about this change.
>>>> >>> > >>>>>
>>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include
>>>> the
>>>> >>> > >>>>>> profiles in
>>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions
>>>> (including the
>>>> >>> > >> default
>>>> >>> > >>>>>> version of flink-shaded).
>>>> >>> > >>>>>>
>>>> >>> > >>>>>> As far as I can tell, the next steps would be:
>>>> >>> > >>>>>>
>>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
>>>> profiles
>>>> >>> > >>>>>> should also
>>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
>>>> >>> > >>>>>>
>>>> >>> > >>>>>> @Chesnay: is this a correct summary?
>>>> >>> > >>>>>>
>>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
>>>> >>> upside, we
>>>> >>> > >>>>>> might
>>>> >>> > >>>>>> get some additional feedback until the 1.10 release with
>>>> these
>>>> >>> > >>>>>> profiles in
>>>> >>> > >>>>>> case users make use of them with 1.8/1.9.
>>>> >>> > >>>>>>
>>>> >>> > >>>>>> – Ufuk
>>>> >>> > >>>>>>
>>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>>>> >>> > [hidden email]
>>>> >>> > >>>>>> wrote:
>>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2;
>>>> on
>>>> >>> master
>>>> >>> > >>>>>>> (and
>>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>>>> >>> > >>>>>>>
>>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we
>>>> usually do
>>>> >>> not
>>>> >>> > >> bump
>>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term
>>>> change
>>>> >>> > that
>>>> >>> > >> we
>>>> >>> > >>>>>>> aren't allowing to mature properly.
>>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further
>>>> away,
>>>> >>> hence
>>>> >>> > no
>>>> >>> > >>>>>>> opt-in.
>>>> >>> > >>>>>>>
>>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades
>>>> being more
>>>> >>> > >> common
>>>> >>> > >>>>>>> in the future.
>>>> >>> > >>>>>>>
>>>> >>> > >>>>>>> We can certainly support every jackson version that fixes
>>>> these
>>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a
>>>> different
>>>> >>> > >> version
>>>> >>> > >>>>>>> (that hopefully includes the fixes).
>>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that
>>>> may
>>>> >>> or may
>>>> >>> > >> not
>>>> >>> > >>>>>>> be feasible.
>>>> >>> > >>>>>>>
>>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>>>> >>> > >>>>>>>> Hi Chesnay,
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
>>>> >>> Really a
>>>> >>> > >> good
>>>> >>> > >>>>>> job!
>>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>>>> >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities,
>>>> >>> some of
>>>> >>> > >> them
>>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my
>>>> point of
>>>> >>> > view,
>>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I
>>>> would
>>>> >>> like
>>>> >>> > >> to
>>>> >>> > >>>>>>>> trust your judgment as you are more professional at this
>>>> >>> problem.
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9
>>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea.
>>>> Another
>>>> >>> > question
>>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson
>>>> versions
>>>> >>> that
>>>> >>> > >> have
>>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan
>>>> to
>>>> >>> > support
>>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As
>>>> an
>>>> >>> > option,
>>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems
>>>> using
>>>> >>> the
>>>> >>> > new
>>>> >>> > >>>>>>>> version. Of course, we will try our best to make the new
>>>> >>> release
>>>> >>> > out
>>>> >>> > >>>>>>>> of question.
>>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it
>>>> will
>>>> >>> make
>>>> >>> > our
>>>> >>> > >>>>>>>> build unlikely convergence as more and more build options
>>>> >>> will be
>>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>> What do you think?
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>> Best, Hequn
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>>>> >>> > >> [hidden email]
>>>> >>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>        So here's the state of things:
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>        The master of flink-shaded now uses jackson
>>>> 2.10.1,
>>>> >>> which
>>>> >>> > >>>>>>>>        eliminates a whole category of security
>>>> >>> vulnerabilities.
>>>> >>> > >>>>>>>>        The flink master works perfectly fine with that
>>>> >>> version;
>>>> >>> > 1.9
>>>> >>> > >> will
>>>> >>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>>>> >>> adjustment.
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>        Hence, there may be value in first doing a
>>>> flink-shaded
>>>> >>> > >>>>>>>> release so
>>>> >>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3
>>>> and
>>>> >>> 1.9.2 .
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>        As for other jackson dependencies (coming from
>>>> calcite,
>>>> >>> > kafka,
>>>> >>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of
>>>> master
>>>> >>> > >> yesterday
>>>> >>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1,
>>>> and they
>>>> >>> > >> passed. I
>>>> >>> > >>>>>>>>        will open a PR soon-ish for making this change on
>>>> >>> master.
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>        The question now is whether we want to backport
>>>> this
>>>> >>> > change to
>>>> >>> > >>>>>>>>        1.8/1.9 .
>>>> >>> > >>>>>>>>        Some code paths /may /not be covered by our
>>>> tests, and
>>>> >>> > >> transitive
>>>> >>> > >>>>>>>>        jackson users /might /run into issues.
>>>> >>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>>>> >>> upgrade,
>>>> >>> > by
>>>> >>> > >>>>>>>>        adding a separate profile that bumps the
>>>> versions. This
>>>> >>> > would
>>>> >>> > >>>>>>>>        present users/providers who are concerned about
>>>> the
>>>> >>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk of
>>>> >>> /some
>>>> >>> > >> /things
>>>> >>> > >>>>>>>>        /maybe /not working.
>>>> >>> > >>>>>>>>
>>>> >>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>>>> >>> > >>>>>>>>>        Hi Chesnay, Jincheng
>>>> >>> > >>>>>>>>>
>>>> >>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>>>> >>> > >>>>>>>>>        Thanks a lot for providing the information about
>>>> the
>>>> >>> > security
>>>> >>> > >>>>>>>>>        vulnerabilities! @Chesnay
>>>> >>> > >>>>>>>>>
>>>> >>> > >>>>>>>>>        Best, Hequn
>>>> >>> > >>>>>>>>>
>>>> >>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>>>> >>> > >>>>>> [hidden email]> <mailto:[hidden email]
>>>> >
>>>> >>> > >>>>>>>>>        wrote:
>>>> >>> > >>>>>>>>>
>>>> >>> > >>>>>>>>>>        +1 for try to eliminate the security
>>>> vulnerabilities.
>>>> >>> > Great
>>>> >>> > >>>>>> thanks for
>>>> >>> > >>>>>>>>>>        doing this important work, Chesnay!
>>>> >>> > >>>>>>>>>>        What do you think Hequn ?
>>>> >>> > >>>>>>>>>>
>>>> >>> > >>>>>>>>>>        Best,
>>>> >>> > >>>>>>>>>>        Jincheng
>>>> >>> > >>>>>>>>>>
>>>> >>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>>>> >>> > >>>>>>>>>> <mailto:[hidden email]>
>>>> >>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>>>> >>> > >>>>>>>>>>>        It would be great if you could give me a day
>>>> or 2 to
>>>> >>> > check
>>>> >>> > >> how
>>>> >>> > >>>>>> easy it
>>>> >>> > >>>>>>>>>>>        would be to bump the various jackson
>>>> dependencies to
>>>> >>> > >>>>>>>>>>> eliminate a
>>>> >>> > >>>>>> few
>>>> >>> > >>>>>>>>>>>        security vulnerabilities.
>>>> >>> > >>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>>>> >>> > >>>>>>>>>>>>        Hi Flink devs,
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        It has been more than 2 months since the 1.8.2
>>>> >>> > released.
>>>> >>> > >> So,
>>>> >>> > >>>>>> What do
>>>> >>> > >>>>>>>>>>        you
>>>> >>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        We already have many important bug fixes in
>>>> the
>>>> >>> > >> release-1.8
>>>> >>> > >>>>>> branch (29
>>>> >>> > >>>>>>>>>>>>        resolved issues).
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        Most notable fixes are:
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't
>>>> give
>>>> >>> up
>>>> >>> > >>>>>>>>>>>> leadership
>>>> >>> > >>>>>> when AM
>>>> >>> > >>>>>>>>>>>        is
>>>> >>> > >>>>>>>>>>>>        shut down
>>>> >>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>>>> >>> JobMaster.disconnectTaskManager
>>>> >>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo
>>>> should
>>>> >>> > >> consider
>>>> >>> > >>>>>>>>>>        fieldsNames
>>>> >>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires
>>>> Too
>>>> >>> Many
>>>> >>> > >>>>>>>>>>>> Containers
>>>> >>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with
>>>> the same
>>>> >>> > >>>>>> AllocationID leads
>>>> >>> > >>>>>>>>>>        to
>>>> >>> > >>>>>>>>>>>>        inconsistent slot table
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        Furthermore, the following critical issues is
>>>> in
>>>> >>> > progress,
>>>> >>> > >>>>>> maybe we can
>>>> >>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks
>>>> the
>>>> >>> > >>>>>> YarnResourceManager's
>>>> >>> > >>>>>>>>>>>        main
>>>> >>> > >>>>>>>>>>>>        thread
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        Please let me know what you think?
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>>>>>>>>>>>        Best,
>>>> >>> > >>>>>>>>>>>>        Jincheng
>>>> >>> > >>>>>>>>>>>>
>>>> >>> > >>
>>>> >>> >
>>>> >>> >
>>>> >>>
>>>> >>
>>>>
>>>

[Hequn Cheng]

Hi Jincheng,

Thanks a lot for your timely help. I'm on my way to the release.

Best, Hequn

On Wed, Nov 27, 2019 at 7:36 AM jincheng sun <[hidden email]>
wrote:

> Hi Hequn,
>
> Thank you for your great job! Looking forward the first RC of 1.8.3 !
> BTW: The version of 1.8.4 already created here:
> https://issues.apache.org/jira/projects/FLINK/versions/12346552
>
> Best,
> Jincheng
>
> Hequn Cheng <[hidden email]> 于2019年11月26日周二 下午8:18写道：
>
>> Hi all,
>>
>> I would like to share with you that all blockers are resolved now. If
>> there are no more critical issues, I will create the first RC tomorrow and
>> vote on it directly.
>> Hope everything goes well!
>>
>> Thank you all for the help of fixing, reviewing, driving and discussions!
>>
>> Best, Hequn
>>
>> On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> @Ufuk Celebi <[hidden email]> Hi, we are very close now. There is one
>>> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>)
>>> left that I want to double-check with you guys. Once this is done, we can
>>> create the first RC. I already have some minor comments in the PR
>>> <https://github.com/apache/flink/pull/10195>.
>>>
>>> @Zhu Zhu <[hidden email]> Glad to hear that it is not a blocker.
>>> Thank you.
>>>
>>> Best, Hequn
>>>
>>> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <[hidden email]> wrote:
>>>
>>>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you
>>>> can go ahead and create the first RC. :-)
>>>>
>>>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <[hidden email]> wrote:
>>>>
>>>>> Hi Hequn,
>>>>>
>>>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
>>>>> Given that this fix is for batch job only and batch is not very good in
>>>>> 1.8, I think it is a not blocker of release 1.8.3.
>>>>> So just don't be blocked by it and feel free to cut the RC when other
>>>>> blocking issues are resolved.
>>>>>
>>>>> Thanks,
>>>>> Zhu Zhu
>>>>>
>>>>> Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：
>>>>>
>>>>> > Hi Zhu Zhu,
>>>>> >
>>>>> > Thanks a lot for letting us know!
>>>>> > We can't cut the first RC right now due to the wait of the
>>>>> flink-shade
>>>>> > release, so go ahead.
>>>>> >
>>>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it
>>>>> once the
>>>>> > release of flink-shade is done,
>>>>> > but I will try my best to have it in 1.8.3. Hope we can get it on
>>>>> board on
>>>>> > time. :)
>>>>> >
>>>>> > Best, Hequn
>>>>> >
>>>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
>>>>> >
>>>>> >> Hi Jincheng & Hequn
>>>>> >>
>>>>> >> Thanks for driving the releasing of 1.8.3.
>>>>> >>
>>>>> >> I am now working on FLINK-14735. The fix avoids duplicated input
>>>>> >> checking when scheduling ALL-to-ALL
>>>>> >> connected downstream consumers with ALL input constraints. The
>>>>> duplicated
>>>>> >> checking can cause severe
>>>>> >> performance issues for large scale jobs. So I hope the fix could be
>>>>> >> released with 1.8.3.
>>>>> >>
>>>>> >> The fix is already merged into master, and is now in the process of
>>>>> >> backporting to 1.8.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Zhu Zhu
>>>>> >>
>>>>> >> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>>>>> >>
>>>>> >>> Thanks Chesnay.
>>>>> >>>
>>>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the
>>>>> Jackson
>>>>> >>> version bump and leave those for a future release. Realistically,
>>>>> the
>>>>> >>> flink-shaded release will take until mid next week or end of next
>>>>> week.
>>>>> >>> But
>>>>> >>> please correct me if you think that it should not take that long
>>>>> or it's
>>>>> >>> OK
>>>>> >>> to block the 1.8.3 release on the flink-shaded release.
>>>>> >>>
>>>>> >>> – Ufuk
>>>>> >>>
>>>>> >>>
>>>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <
>>>>> [hidden email]>
>>>>> >>> wrote:
>>>>> >>>
>>>>> >>> > I've kicked off a discussion about the next flink-shaded
>>>>> release, and
>>>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>>>>> >>> >
>>>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>>>>> >>> > > That's great, thank you very much! Ideally, we can kick off the
>>>>> >>> release
>>>>> >>> > > vote for the first RC of 1.8.3 within next week. :)
>>>>> >>> > >
>>>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
>>>>> [hidden email]
>>>>> >>> >
>>>>> >>> > wrote:
>>>>> >>> > >
>>>>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so
>>>>> we
>>>>> >>> could
>>>>> >>> > >> start the release right away.
>>>>> >>> > >>
>>>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>>>>> >>> > >>> Hi,
>>>>> >>> > >>>
>>>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>>>>> >>> approach
>>>>> >>> > for
>>>>> >>> > >>> 1.8/1.9.
>>>>> >>> > >>> @Ufuk Thank you for the nice summary.
>>>>> >>> > >>>
>>>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a
>>>>> bit to
>>>>> >>> first
>>>>> >>> > >> do a
>>>>> >>> > >>> flink-shaded release.
>>>>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded
>>>>> with
>>>>> >>> > >> upgraded
>>>>> >>> > >>> Jackson?
>>>>> >>> > >>>
>>>>> >>> > >>> Best, Hequn
>>>>> >>> > >>>
>>>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>>>>> >>> [hidden email]>
>>>>> >>> > >> wrote:
>>>>> >>> > >>>> One small modification: the flink-shaded upgrade does not
>>>>> have to
>>>>> >>> be
>>>>> >>> > >>>> part of the profile; since it is only intended for internal
>>>>> use
>>>>> >>> anyway
>>>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this
>>>>> doesn't
>>>>> >>> > break
>>>>> >>> > >>>> anything.
>>>>> >>> > >>>>
>>>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>>>>> >>> > >>>>> Ufuk's summary is correct.
>>>>> >>> > >>>>>
>>>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the
>>>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>>>>> >>> > >>>>> but I have no concerns about this change.
>>>>> >>> > >>>>>
>>>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>>>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include
>>>>> the
>>>>> >>> > >>>>>> profiles in
>>>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions
>>>>> (including the
>>>>> >>> > >> default
>>>>> >>> > >>>>>> version of flink-shaded).
>>>>> >>> > >>>>>>
>>>>> >>> > >>>>>> As far as I can tell, the next steps would be:
>>>>> >>> > >>>>>>
>>>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>>>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>>>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
>>>>> profiles
>>>>> >>> > >>>>>> should also
>>>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded version)
>>>>> >>> > >>>>>>
>>>>> >>> > >>>>>> @Chesnay: is this a correct summary?
>>>>> >>> > >>>>>>
>>>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
>>>>> >>> upside, we
>>>>> >>> > >>>>>> might
>>>>> >>> > >>>>>> get some additional feedback until the 1.10 release with
>>>>> these
>>>>> >>> > >>>>>> profiles in
>>>>> >>> > >>>>>> case users make use of them with 1.8/1.9.
>>>>> >>> > >>>>>>
>>>>> >>> > >>>>>> – Ufuk
>>>>> >>> > >>>>>>
>>>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>>>>> >>> > [hidden email]
>>>>> >>> > >>>>>> wrote:
>>>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 /
>>>>> 1.9.2; on
>>>>> >>> master
>>>>> >>> > >>>>>>> (and
>>>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>>>>> >>> > >>>>>>>
>>>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we
>>>>> usually do
>>>>> >>> not
>>>>> >>> > >> bump
>>>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term
>>>>> change
>>>>> >>> > that
>>>>> >>> > >> we
>>>>> >>> > >>>>>>> aren't allowing to mature properly.
>>>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further
>>>>> away,
>>>>> >>> hence
>>>>> >>> > no
>>>>> >>> > >>>>>>> opt-in.
>>>>> >>> > >>>>>>>
>>>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades
>>>>> being more
>>>>> >>> > >> common
>>>>> >>> > >>>>>>> in the future.
>>>>> >>> > >>>>>>>
>>>>> >>> > >>>>>>> We can certainly support every jackson version that
>>>>> fixes these
>>>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a
>>>>> different
>>>>> >>> > >> version
>>>>> >>> > >>>>>>> (that hopefully includes the fixes).
>>>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that
>>>>> may
>>>>> >>> or may
>>>>> >>> > >> not
>>>>> >>> > >>>>>>> be feasible.
>>>>> >>> > >>>>>>>
>>>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>>>>> >>> > >>>>>>>> Hi Chesnay,
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
>>>>> >>> Really a
>>>>> >>> > >> good
>>>>> >>> > >>>>>> job!
>>>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>>>>> >>> > >>>>>>>> I had taken a quick look at the security
>>>>> vulnerabilities,
>>>>> >>> some of
>>>>> >>> > >> them
>>>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my
>>>>> point of
>>>>> >>> > view,
>>>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I
>>>>> would
>>>>> >>> like
>>>>> >>> > >> to
>>>>> >>> > >>>>>>>> trust your judgment as you are more professional at this
>>>>> >>> problem.
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9
>>>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea.
>>>>> Another
>>>>> >>> > question
>>>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson
>>>>> versions
>>>>> >>> that
>>>>> >>> > >> have
>>>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only
>>>>> plan to
>>>>> >>> > support
>>>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade.
>>>>> As an
>>>>> >>> > option,
>>>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems
>>>>> using
>>>>> >>> the
>>>>> >>> > new
>>>>> >>> > >>>>>>>> version. Of course, we will try our best to make the new
>>>>> >>> release
>>>>> >>> > out
>>>>> >>> > >>>>>>>> of question.
>>>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it
>>>>> will
>>>>> >>> make
>>>>> >>> > our
>>>>> >>> > >>>>>>>> build unlikely convergence as more and more build
>>>>> options
>>>>> >>> will be
>>>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this one.
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>> What do you think?
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>> Best, Hequn
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>>>>> >>> > >> [hidden email]
>>>>> >>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>        So here's the state of things:
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>        The master of flink-shaded now uses jackson
>>>>> 2.10.1,
>>>>> >>> which
>>>>> >>> > >>>>>>>>        eliminates a whole category of security
>>>>> >>> vulnerabilities.
>>>>> >>> > >>>>>>>>        The flink master works perfectly fine with that
>>>>> >>> version;
>>>>> >>> > 1.9
>>>>> >>> > >> will
>>>>> >>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>>>>> >>> adjustment.
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>        Hence, there may be value in first doing a
>>>>> flink-shaded
>>>>> >>> > >>>>>>>> release so
>>>>> >>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3
>>>>> and
>>>>> >>> 1.9.2 .
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>        As for other jackson dependencies (coming from
>>>>> calcite,
>>>>> >>> > kafka,
>>>>> >>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests of
>>>>> master
>>>>> >>> > >> yesterday
>>>>> >>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1,
>>>>> and they
>>>>> >>> > >> passed. I
>>>>> >>> > >>>>>>>>        will open a PR soon-ish for making this change on
>>>>> >>> master.
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>        The question now is whether we want to backport
>>>>> this
>>>>> >>> > change to
>>>>> >>> > >>>>>>>>        1.8/1.9 .
>>>>> >>> > >>>>>>>>        Some code paths /may /not be covered by our
>>>>> tests, and
>>>>> >>> > >> transitive
>>>>> >>> > >>>>>>>>        jackson users /might /run into issues.
>>>>> >>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>>>>> >>> upgrade,
>>>>> >>> > by
>>>>> >>> > >>>>>>>>        adding a separate profile that bumps the
>>>>> versions. This
>>>>> >>> > would
>>>>> >>> > >>>>>>>>        present users/providers who are concerned about
>>>>> the
>>>>> >>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk
>>>>> of
>>>>> >>> /some
>>>>> >>> > >> /things
>>>>> >>> > >>>>>>>>        /maybe /not working.
>>>>> >>> > >>>>>>>>
>>>>> >>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>>>>> >>> > >>>>>>>>>        Hi Chesnay, Jincheng
>>>>> >>> > >>>>>>>>>
>>>>> >>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>>>>> >>> > >>>>>>>>>        Thanks a lot for providing the information
>>>>> about the
>>>>> >>> > security
>>>>> >>> > >>>>>>>>>        vulnerabilities! @Chesnay
>>>>> >>> > >>>>>>>>>
>>>>> >>> > >>>>>>>>>        Best, Hequn
>>>>> >>> > >>>>>>>>>
>>>>> >>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>>>>> >>> > >>>>>> [hidden email]> <mailto:
>>>>> [hidden email]>
>>>>> >>> > >>>>>>>>>        wrote:
>>>>> >>> > >>>>>>>>>
>>>>> >>> > >>>>>>>>>>        +1 for try to eliminate the security
>>>>> vulnerabilities.
>>>>> >>> > Great
>>>>> >>> > >>>>>> thanks for
>>>>> >>> > >>>>>>>>>>        doing this important work, Chesnay!
>>>>> >>> > >>>>>>>>>>        What do you think Hequn ?
>>>>> >>> > >>>>>>>>>>
>>>>> >>> > >>>>>>>>>>        Best,
>>>>> >>> > >>>>>>>>>>        Jincheng
>>>>> >>> > >>>>>>>>>>
>>>>> >>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>>>>> >>> > >>>>>>>>>> <mailto:[hidden email]>
>>>>> >>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>>>>> >>> > >>>>>>>>>>>        It would be great if you could give me a day
>>>>> or 2 to
>>>>> >>> > check
>>>>> >>> > >> how
>>>>> >>> > >>>>>> easy it
>>>>> >>> > >>>>>>>>>>>        would be to bump the various jackson
>>>>> dependencies to
>>>>> >>> > >>>>>>>>>>> eliminate a
>>>>> >>> > >>>>>> few
>>>>> >>> > >>>>>>>>>>>        security vulnerabilities.
>>>>> >>> > >>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>>>>> >>> > >>>>>>>>>>>>        Hi Flink devs,
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        It has been more than 2 months since the
>>>>> 1.8.2
>>>>> >>> > released.
>>>>> >>> > >> So,
>>>>> >>> > >>>>>> What do
>>>>> >>> > >>>>>>>>>>        you
>>>>> >>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        We already have many important bug fixes in
>>>>> the
>>>>> >>> > >> release-1.8
>>>>> >>> > >>>>>> branch (29
>>>>> >>> > >>>>>>>>>>>>        resolved issues).
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        Most notable fixes are:
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers don't
>>>>> give
>>>>> >>> up
>>>>> >>> > >>>>>>>>>>>> leadership
>>>>> >>> > >>>>>> when AM
>>>>> >>> > >>>>>>>>>>>        is
>>>>> >>> > >>>>>>>>>>>>        shut down
>>>>> >>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>>>>> >>> JobMaster.disconnectTaskManager
>>>>> >>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in RowTypeInfo
>>>>> should
>>>>> >>> > >> consider
>>>>> >>> > >>>>>>>>>>        fieldsNames
>>>>> >>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager Acquires
>>>>> Too
>>>>> >>> Many
>>>>> >>> > >>>>>>>>>>>> Containers
>>>>> >>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with
>>>>> the same
>>>>> >>> > >>>>>> AllocationID leads
>>>>> >>> > >>>>>>>>>>        to
>>>>> >>> > >>>>>>>>>>>>        inconsistent slot table
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        Furthermore, the following critical issues
>>>>> is in
>>>>> >>> > progress,
>>>>> >>> > >>>>>> maybe we can
>>>>> >>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor blocks
>>>>> the
>>>>> >>> > >>>>>> YarnResourceManager's
>>>>> >>> > >>>>>>>>>>>        main
>>>>> >>> > >>>>>>>>>>>>        thread
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        Please let me know what you think?
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>>>>>>>>>>>        Best,
>>>>> >>> > >>>>>>>>>>>>        Jincheng
>>>>> >>> > >>>>>>>>>>>>
>>>>> >>> > >>
>>>>> >>> >
>>>>> >>> >
>>>>> >>>
>>>>> >>
>>>>>
>>>>

[Hequn Cheng]

Hi all,

Thank you all for fixing issues of 1.8.3 release!
The VOTE mail thread of the first RC of 1.8.3 has already been brought up.
I would appreciate it if you can help to check the release and VOTE for the
RC1.

Thanks,
Hequn

On Wed, Nov 27, 2019 at 11:36 AM Hequn Cheng <[hidden email]> wrote:

> Hi Jincheng,
>
> Thanks a lot for your timely help. I'm on my way to the release.
>
> Best, Hequn
>
> On Wed, Nov 27, 2019 at 7:36 AM jincheng sun <[hidden email]>
> wrote:
>
>> Hi Hequn,
>>
>> Thank you for your great job! Looking forward the first RC of 1.8.3 !
>> BTW: The version of 1.8.4 already created here:
>> https://issues.apache.org/jira/projects/FLINK/versions/12346552
>>
>> Best,
>> Jincheng
>>
>> Hequn Cheng <[hidden email]> 于2019年11月26日周二 下午8:18写道：
>>
>>> Hi all,
>>>
>>> I would like to share with you that all blockers are resolved now. If
>>> there are no more critical issues, I will create the first RC tomorrow and
>>> vote on it directly.
>>> Hope everything goes well!
>>>
>>> Thank you all for the help of fixing, reviewing, driving and discussions!
>>>
>>> Best, Hequn
>>>
>>> On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <[hidden email]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> @Ufuk Celebi <[hidden email]> Hi, we are very close now. There is one
>>>> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>)
>>>> left that I want to double-check with you guys. Once this is done, we can
>>>> create the first RC. I already have some minor comments in the PR
>>>> <https://github.com/apache/flink/pull/10195>.
>>>>
>>>> @Zhu Zhu <[hidden email]> Glad to hear that it is not a blocker.
>>>> Thank you.
>>>>
>>>> Best, Hequn
>>>>
>>>> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <[hidden email]> wrote:
>>>>
>>>>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think
>>>>> you can go ahead and create the first RC. :-)
>>>>>
>>>>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <[hidden email]> wrote:
>>>>>
>>>>>> Hi Hequn,
>>>>>>
>>>>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
>>>>>> Given that this fix is for batch job only and batch is not very good
>>>>>> in
>>>>>> 1.8, I think it is a not blocker of release 1.8.3.
>>>>>> So just don't be blocked by it and feel free to cut the RC when other
>>>>>> blocking issues are resolved.
>>>>>>
>>>>>> Thanks,
>>>>>> Zhu Zhu
>>>>>>
>>>>>> Hequn Cheng <[hidden email]> 于2019年11月23日周六 下午9:08写道：
>>>>>>
>>>>>> > Hi Zhu Zhu,
>>>>>> >
>>>>>> > Thanks a lot for letting us know!
>>>>>> > We can't cut the first RC right now due to the wait of the
>>>>>> flink-shade
>>>>>> > release, so go ahead.
>>>>>> >
>>>>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it
>>>>>> once the
>>>>>> > release of flink-shade is done,
>>>>>> > but I will try my best to have it in 1.8.3. Hope we can get it on
>>>>>> board on
>>>>>> > time. :)
>>>>>> >
>>>>>> > Best, Hequn
>>>>>> >
>>>>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <[hidden email]> wrote:
>>>>>> >
>>>>>> >> Hi Jincheng & Hequn
>>>>>> >>
>>>>>> >> Thanks for driving the releasing of 1.8.3.
>>>>>> >>
>>>>>> >> I am now working on FLINK-14735. The fix avoids duplicated input
>>>>>> >> checking when scheduling ALL-to-ALL
>>>>>> >> connected downstream consumers with ALL input constraints. The
>>>>>> duplicated
>>>>>> >> checking can cause severe
>>>>>> >> performance issues for large scale jobs. So I hope the fix could be
>>>>>> >> released with 1.8.3.
>>>>>> >>
>>>>>> >> The fix is already merged into master, and is now in the process of
>>>>>> >> backporting to 1.8.
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> Zhu Zhu
>>>>>> >>
>>>>>> >> Ufuk Celebi <[hidden email]> 于2019年11月15日周五 下午11:54写道：
>>>>>> >>
>>>>>> >>> Thanks Chesnay.
>>>>>> >>>
>>>>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the
>>>>>> Jackson
>>>>>> >>> version bump and leave those for a future release. Realistically,
>>>>>> the
>>>>>> >>> flink-shaded release will take until mid next week or end of next
>>>>>> week.
>>>>>> >>> But
>>>>>> >>> please correct me if you think that it should not take that long
>>>>>> or it's
>>>>>> >>> OK
>>>>>> >>> to block the 1.8.3 release on the flink-shaded release.
>>>>>> >>>
>>>>>> >>> – Ufuk
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <
>>>>>> [hidden email]>
>>>>>> >>> wrote:
>>>>>> >>>
>>>>>> >>> > I've kicked off a discussion about the next flink-shaded
>>>>>> release, and
>>>>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>>>>>> >>> >
>>>>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>>>>>> >>> > > That's great, thank you very much! Ideally, we can kick off
>>>>>> the
>>>>>> >>> release
>>>>>> >>> > > vote for the first RC of 1.8.3 within next week. :)
>>>>>> >>> > >
>>>>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
>>>>>> [hidden email]
>>>>>> >>> >
>>>>>> >>> > wrote:
>>>>>> >>> > >
>>>>>> >>> > >> I'm not aware of any more planned changes to flink-shaded;
>>>>>> so we
>>>>>> >>> could
>>>>>> >>> > >> start the release right away.
>>>>>> >>> > >>
>>>>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>>>>>> >>> > >>> Hi,
>>>>>> >>> > >>>
>>>>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>>>>>> >>> approach
>>>>>> >>> > for
>>>>>> >>> > >>> 1.8/1.9.
>>>>>> >>> > >>> @Ufuk Thank you for the nice summary.
>>>>>> >>> > >>>
>>>>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a
>>>>>> bit to
>>>>>> >>> first
>>>>>> >>> > >> do a
>>>>>> >>> > >>> flink-shaded release.
>>>>>> >>> > >>> BTW, @chesnay when would we plan to release the
>>>>>> flink-shaded with
>>>>>> >>> > >> upgraded
>>>>>> >>> > >>> Jackson?
>>>>>> >>> > >>>
>>>>>> >>> > >>> Best, Hequn
>>>>>> >>> > >>>
>>>>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>>>>>> >>> [hidden email]>
>>>>>> >>> > >> wrote:
>>>>>> >>> > >>>> One small modification: the flink-shaded upgrade does not
>>>>>> have to
>>>>>> >>> be
>>>>>> >>> > >>>> part of the profile; since it is only intended for
>>>>>> internal use
>>>>>> >>> anyway
>>>>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this
>>>>>> doesn't
>>>>>> >>> > break
>>>>>> >>> > >>>> anything.
>>>>>> >>> > >>>>
>>>>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote:
>>>>>> >>> > >>>>> Ufuk's summary is correct.
>>>>>> >>> > >>>>>
>>>>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the
>>>>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson,
>>>>>> >>> > >>>>> but I have no concerns about this change.
>>>>>> >>> > >>>>>
>>>>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote:
>>>>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to
>>>>>> include the
>>>>>> >>> > >>>>>> profiles in
>>>>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions
>>>>>> (including the
>>>>>> >>> > >> default
>>>>>> >>> > >>>>>> version of flink-shaded).
>>>>>> >>> > >>>>>>
>>>>>> >>> > >>>>>> As far as I can tell, the next steps would be:
>>>>>> >>> > >>>>>>
>>>>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson
>>>>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master
>>>>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
>>>>>> profiles
>>>>>> >>> > >>>>>> should also
>>>>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded
>>>>>> version)
>>>>>> >>> > >>>>>>
>>>>>> >>> > >>>>>> @Chesnay: is this a correct summary?
>>>>>> >>> > >>>>>>
>>>>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an
>>>>>> >>> upside, we
>>>>>> >>> > >>>>>> might
>>>>>> >>> > >>>>>> get some additional feedback until the 1.10 release with
>>>>>> these
>>>>>> >>> > >>>>>> profiles in
>>>>>> >>> > >>>>>> case users make use of them with 1.8/1.9.
>>>>>> >>> > >>>>>>
>>>>>> >>> > >>>>>> – Ufuk
>>>>>> >>> > >>>>>>
>>>>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <
>>>>>> >>> > [hidden email]
>>>>>> >>> > >>>>>> wrote:
>>>>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 /
>>>>>> 1.9.2; on
>>>>>> >>> master
>>>>>> >>> > >>>>>>> (and
>>>>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in.
>>>>>> >>> > >>>>>>>
>>>>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we
>>>>>> usually do
>>>>>> >>> not
>>>>>> >>> > >> bump
>>>>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a
>>>>>> short-term change
>>>>>> >>> > that
>>>>>> >>> > >> we
>>>>>> >>> > >>>>>>> aren't allowing to mature properly.
>>>>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further
>>>>>> away,
>>>>>> >>> hence
>>>>>> >>> > no
>>>>>> >>> > >>>>>>> opt-in.
>>>>>> >>> > >>>>>>>
>>>>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades
>>>>>> being more
>>>>>> >>> > >> common
>>>>>> >>> > >>>>>>> in the future.
>>>>>> >>> > >>>>>>>
>>>>>> >>> > >>>>>>> We can certainly support every jackson version that
>>>>>> fixes these
>>>>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a
>>>>>> different
>>>>>> >>> > >> version
>>>>>> >>> > >>>>>>> (that hopefully includes the fixes).
>>>>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but
>>>>>> that may
>>>>>> >>> or may
>>>>>> >>> > >> not
>>>>>> >>> > >>>>>>> be feasible.
>>>>>> >>> > >>>>>>>
>>>>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote:
>>>>>> >>> > >>>>>>>> Hi Chesnay,
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master.
>>>>>> >>> Really a
>>>>>> >>> > >> good
>>>>>> >>> > >>>>>> job!
>>>>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9
>>>>>> >>> > >>>>>>>> I had taken a quick look at the security
>>>>>> vulnerabilities,
>>>>>> >>> some of
>>>>>> >>> > >> them
>>>>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my
>>>>>> point of
>>>>>> >>> > view,
>>>>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However,
>>>>>> I would
>>>>>> >>> like
>>>>>> >>> > >> to
>>>>>> >>> > >>>>>>>> trust your judgment as you are more professional at
>>>>>> this
>>>>>> >>> problem.
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9
>>>>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea.
>>>>>> Another
>>>>>> >>> > question
>>>>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson
>>>>>> versions
>>>>>> >>> that
>>>>>> >>> > >> have
>>>>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only
>>>>>> plan to
>>>>>> >>> > support
>>>>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade.
>>>>>> As an
>>>>>> >>> > option,
>>>>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems
>>>>>> using
>>>>>> >>> the
>>>>>> >>> > new
>>>>>> >>> > >>>>>>>> version. Of course, we will try our best to make the
>>>>>> new
>>>>>> >>> release
>>>>>> >>> > out
>>>>>> >>> > >>>>>>>> of question.
>>>>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it
>>>>>> will
>>>>>> >>> make
>>>>>> >>> > our
>>>>>> >>> > >>>>>>>> build unlikely convergence as more and more build
>>>>>> options
>>>>>> >>> will be
>>>>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this
>>>>>> one.
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>> What do you think?
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>> Best, Hequn
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <
>>>>>> >>> > >> [hidden email]
>>>>>> >>> > >>>>>>>> <mailto:[hidden email]>> wrote:
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>        So here's the state of things:
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>        The master of flink-shaded now uses jackson
>>>>>> 2.10.1,
>>>>>> >>> which
>>>>>> >>> > >>>>>>>>        eliminates a whole category of security
>>>>>> >>> vulnerabilities.
>>>>>> >>> > >>>>>>>>        The flink master works perfectly fine with that
>>>>>> >>> version;
>>>>>> >>> > 1.9
>>>>>> >>> > >> will
>>>>>> >>> > >>>>>>>>        likely do so too and 1.8 would require a minor
>>>>>> >>> adjustment.
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>        Hence, there may be value in first doing a
>>>>>> flink-shaded
>>>>>> >>> > >>>>>>>> release so
>>>>>> >>> > >>>>>>>>        we can eliminate these vulnerabilities in 1.8.3
>>>>>> and
>>>>>> >>> 1.9.2 .
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>        As for other jackson dependencies (coming from
>>>>>> calcite,
>>>>>> >>> > kafka,
>>>>>> >>> > >>>>>>>>        kinesis), I ran the unit and end-to-end tests
>>>>>> of master
>>>>>> >>> > >> yesterday
>>>>>> >>> > >>>>>>>>        will /all /jackson dependencies set to 2.10.1,
>>>>>> and they
>>>>>> >>> > >> passed. I
>>>>>> >>> > >>>>>>>>        will open a PR soon-ish for making this change
>>>>>> on
>>>>>> >>> master.
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>        The question now is whether we want to backport
>>>>>> this
>>>>>> >>> > change to
>>>>>> >>> > >>>>>>>>        1.8/1.9 .
>>>>>> >>> > >>>>>>>>        Some code paths /may /not be covered by our
>>>>>> tests, and
>>>>>> >>> > >> transitive
>>>>>> >>> > >>>>>>>>        jackson users /might /run into issues.
>>>>>> >>> > >>>>>>>>        Alternatively, we could set this up as an opt-in
>>>>>> >>> upgrade,
>>>>>> >>> > by
>>>>>> >>> > >>>>>>>>        adding a separate profile that bumps the
>>>>>> versions. This
>>>>>> >>> > would
>>>>>> >>> > >>>>>>>>        present users/providers who are concerned about
>>>>>> the
>>>>>> >>> > >>>>>>>>        vulnerabilities an easy workaround, at the risk
>>>>>> of
>>>>>> >>> /some
>>>>>> >>> > >> /things
>>>>>> >>> > >>>>>>>>        /maybe /not working.
>>>>>> >>> > >>>>>>>>
>>>>>> >>> > >>>>>>>>        On 14/11/2019 03:16, Hequn Cheng wrote:
>>>>>> >>> > >>>>>>>>>        Hi Chesnay, Jincheng
>>>>>> >>> > >>>>>>>>>
>>>>>> >>> > >>>>>>>>>        Sure, I think it's good to have these fixes.
>>>>>> >>> > >>>>>>>>>        Thanks a lot for providing the information
>>>>>> about the
>>>>>> >>> > security
>>>>>> >>> > >>>>>>>>>        vulnerabilities! @Chesnay
>>>>>> >>> > >>>>>>>>>
>>>>>> >>> > >>>>>>>>>        Best, Hequn
>>>>>> >>> > >>>>>>>>>
>>>>>> >>> > >>>>>>>>>        On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
>>>>>> >>> > >>>>>> [hidden email]> <mailto:
>>>>>> [hidden email]>
>>>>>> >>> > >>>>>>>>>        wrote:
>>>>>> >>> > >>>>>>>>>
>>>>>> >>> > >>>>>>>>>>        +1 for try to eliminate the security
>>>>>> vulnerabilities.
>>>>>> >>> > Great
>>>>>> >>> > >>>>>> thanks for
>>>>>> >>> > >>>>>>>>>>        doing this important work, Chesnay!
>>>>>> >>> > >>>>>>>>>>        What do you think Hequn ?
>>>>>> >>> > >>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>        Best,
>>>>>> >>> > >>>>>>>>>>        Jincheng
>>>>>> >>> > >>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>        Chesnay Schepler<[hidden email]>
>>>>>> >>> > >>>>>>>>>> <mailto:[hidden email]>
>>>>>> >>> > >>>>>>     于2019年11月13日周三 下午5:17写道：
>>>>>> >>> > >>>>>>>>>>>        It would be great if you could give me a day
>>>>>> or 2 to
>>>>>> >>> > check
>>>>>> >>> > >> how
>>>>>> >>> > >>>>>> easy it
>>>>>> >>> > >>>>>>>>>>>        would be to bump the various jackson
>>>>>> dependencies to
>>>>>> >>> > >>>>>>>>>>> eliminate a
>>>>>> >>> > >>>>>> few
>>>>>> >>> > >>>>>>>>>>>        security vulnerabilities.
>>>>>> >>> > >>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>        On 09/11/2019 05:10, jincheng sun wrote:
>>>>>> >>> > >>>>>>>>>>>>        Hi Flink devs,
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        It has been more than 2 months since the
>>>>>> 1.8.2
>>>>>> >>> > released.
>>>>>> >>> > >> So,
>>>>>> >>> > >>>>>> What do
>>>>>> >>> > >>>>>>>>>>        you
>>>>>> >>> > >>>>>>>>>>>>        think about releasing Flink 1.8.3 soon?
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        We already have many important bug fixes in
>>>>>> the
>>>>>> >>> > >> release-1.8
>>>>>> >>> > >>>>>> branch (29
>>>>>> >>> > >>>>>>>>>>>>        resolved issues).
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        Most notable fixes are:
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        - FLINK-14010 Dispatcher & JobManagers
>>>>>> don't give
>>>>>> >>> up
>>>>>> >>> > >>>>>>>>>>>> leadership
>>>>>> >>> > >>>>>> when AM
>>>>>> >>> > >>>>>>>>>>>        is
>>>>>> >>> > >>>>>>>>>>>>        shut down
>>>>>> >>> > >>>>>>>>>>>>        - FLINK-14315 NPE with
>>>>>> >>> JobMaster.disconnectTaskManager
>>>>>> >>> > >>>>>>>>>>>>        - FLINK-12848 Method equals() in
>>>>>> RowTypeInfo should
>>>>>> >>> > >> consider
>>>>>> >>> > >>>>>>>>>>        fieldsNames
>>>>>> >>> > >>>>>>>>>>>>        - FLINK-12342 Yarn Resource Manager
>>>>>> Acquires Too
>>>>>> >>> Many
>>>>>> >>> > >>>>>>>>>>>> Containers
>>>>>> >>> > >>>>>>>>>>>>        - FLINK-14589 Redundant slot requests with
>>>>>> the same
>>>>>> >>> > >>>>>> AllocationID leads
>>>>>> >>> > >>>>>>>>>>        to
>>>>>> >>> > >>>>>>>>>>>>        inconsistent slot table
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        Furthermore, the following critical issues
>>>>>> is in
>>>>>> >>> > progress,
>>>>>> >>> > >>>>>> maybe we can
>>>>>> >>> > >>>>>>>>>>>>        wait for it if it is not too much effort.
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        - FLINK-13184 Starting a TaskExecutor
>>>>>> blocks the
>>>>>> >>> > >>>>>> YarnResourceManager's
>>>>>> >>> > >>>>>>>>>>>        main
>>>>>> >>> > >>>>>>>>>>>>        thread
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        Please let me know what you think?
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>>>>>>>>>>>        Best,
>>>>>> >>> > >>>>>>>>>>>>        Jincheng
>>>>>> >>> > >>>>>>>>>>>>
>>>>>> >>> > >>
>>>>>> >>> >
>>>>>> >>> >
>>>>>> >>>
>>>>>> >>
>>>>>>
>>>>>