(DEPRECATED) Apache Flink Mailing List archive.

[DISCUSS] Releasing Apache Flink 1.11.3

Classic

List

Threaded

39 messages Options

Tzu-Li (Gordon) Tai

[DISCUSS] Releasing Apache Flink 1.11.3

Hi,

Xintong and I would like to start a discussion for releasing Flink 1.11.3
soon.

It seems like we already have a few pressing issues that needs to be
included in a new hotfix release:

- Heap-based timers’ restore behaviour is causing a critical recovery
issue for StateFun [1] [2] [3].
- There are several robustness issues for the FLIP-27 new source API,
such as [4]. We already have some users using the FLIP-27 API with 1.11.x,
so it would be important to get those fixes in for 1.11.x as well.

Apart from the issues that are already marked as blocker for 1.11.3 in our
JIRA [5], please let us know in this thread if there is already ongoing
work for other important fixes that we should try to include.

Xintong and I would like to volunteer for managing this release, and will
try to communicate the priority of pending blockers over the next few days.
Since the aforementioned issues are quite critical, we’d like to aim
for a *feature
freeze by the end of next week (Oct. 30th)* and start the release voting
process the week after.
If that is too short of a notice and you might need more time, please let
us know!

Cheers,
Gordon

[1] https://issues.apache.org/jira/browse/FLINK-19692
[2] https://issues.apache.org/jira/browse/FLINK-19741
[3] https://issues.apache.org/jira/browse/FLINK-19748
[4] https://issues.apache.org/jira/browse/FLINK-19717
[5]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3

Till Rohrmann

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Thanks for starting this discussion Gordon. There are over 100 issues which
are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks for
volunteering as our release managers Gordon and Xintong!

Cheers,
Till

On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> Hi,
>
> Xintong and I would like to start a discussion for releasing Flink 1.11.3
> soon.
>
> It seems like we already have a few pressing issues that needs to be
> included in a new hotfix release:
>
> - Heap-based timers’ restore behaviour is causing a critical recovery
> issue for StateFun [1] [2] [3].
> - There are several robustness issues for the FLIP-27 new source API,
> such as [4]. We already have some users using the FLIP-27 API with
> 1.11.x,
> so it would be important to get those fixes in for 1.11.x as well.
>
> Apart from the issues that are already marked as blocker for 1.11.3 in our
> JIRA [5], please let us know in this thread if there is already ongoing
> work for other important fixes that we should try to include.
>
> Xintong and I would like to volunteer for managing this release, and will
> try to communicate the priority of pending blockers over the next few days.
> Since the aforementioned issues are quite critical, we’d like to aim
> for a *feature
> freeze by the end of next week (Oct. 30th)* and start the release voting
> process the week after.
> If that is too short of a notice and you might need more time, please let
> us know!
>
> Cheers,
> Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-19692
> [2] https://issues.apache.org/jira/browse/FLINK-19741
> [3] https://issues.apache.org/jira/browse/FLINK-19748
> [4] https://issues.apache.org/jira/browse/FLINK-19717
> [5]
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>

Xintong Song

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Thanks Gordan for starting this discussion.
My pleasure to help with the release process.

Thank you~

Xintong Song

On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <[hidden email]> wrote:

> Thanks for starting this discussion Gordon. There are over 100 issues
> which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks
> for volunteering as our release managers Gordon and Xintong!
>
> Cheers,
> Till
>
> On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
>> Hi,
>>
>> Xintong and I would like to start a discussion for releasing Flink 1.11.3
>> soon.
>>
>> It seems like we already have a few pressing issues that needs to be
>> included in a new hotfix release:
>>
>> - Heap-based timers’ restore behaviour is causing a critical recovery
>> issue for StateFun [1] [2] [3].
>> - There are several robustness issues for the FLIP-27 new source API,
>> such as [4]. We already have some users using the FLIP-27 API with
>> 1.11.x,
>> so it would be important to get those fixes in for 1.11.x as well.
>>
>> Apart from the issues that are already marked as blocker for 1.11.3 in our
>> JIRA [5], please let us know in this thread if there is already ongoing
>> work for other important fixes that we should try to include.
>>
>> Xintong and I would like to volunteer for managing this release, and will
>> try to communicate the priority of pending blockers over the next few
>> days.
>> Since the aforementioned issues are quite critical, we’d like to aim
>> for a *feature
>> freeze by the end of next week (Oct. 30th)* and start the release voting
>> process the week after.
>> If that is too short of a notice and you might need more time, please let
>> us know!
>>
>> Cheers,
>> Gordon
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-19692
>> [2] https://issues.apache.org/jira/browse/FLINK-19741
>> [3] https://issues.apache.org/jira/browse/FLINK-19748
>> [4] https://issues.apache.org/jira/browse/FLINK-19717
>> [5]
>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>>
>

Yu Li

Re: [DISCUSS] Releasing Apache Flink 1.11.3

+1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
volunteering as our release managers.

Best Regards,
Yu

On Mon, 26 Oct 2020 at 09:45, Xintong Song <[hidden email]> wrote:

> Thanks Gordan for starting this discussion.
> My pleasure to help with the release process.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <[hidden email]>
> wrote:
>
> > Thanks for starting this discussion Gordon. There are over 100 issues
> > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks
> > for volunteering as our release managers Gordon and Xintong!
> >
> > Cheers,
> > Till
> >
> > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <[hidden email]
> >
> > wrote:
> >
> >> Hi,
> >>
> >> Xintong and I would like to start a discussion for releasing Flink
> 1.11.3
> >> soon.
> >>
> >> It seems like we already have a few pressing issues that needs to be
> >> included in a new hotfix release:
> >>
> >> - Heap-based timers’ restore behaviour is causing a critical recovery
> >> issue for StateFun [1] [2] [3].
> >> - There are several robustness issues for the FLIP-27 new source API,
> >> such as [4]. We already have some users using the FLIP-27 API with
> >> 1.11.x,
> >> so it would be important to get those fixes in for 1.11.x as well.
> >>
> >> Apart from the issues that are already marked as blocker for 1.11.3 in
> our
> >> JIRA [5], please let us know in this thread if there is already ongoing
> >> work for other important fixes that we should try to include.
> >>
> >> Xintong and I would like to volunteer for managing this release, and
> will
> >> try to communicate the priority of pending blockers over the next few
> >> days.
> >> Since the aforementioned issues are quite critical, we’d like to aim
> >> for a *feature
> >> freeze by the end of next week (Oct. 30th)* and start the release voting
> >> process the week after.
> >> If that is too short of a notice and you might need more time, please
> let
> >> us know!
> >>
> >> Cheers,
> >> Gordon
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> >> [5]
> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >>
> >
>

Kostas Kloudas-5

Re: [DISCUSS] Releasing Apache Flink 1.11.3

+1 for releasing Flink 1.11.3 as it contains a number of important
fixes and thanks Gordon and Xintong for volunteering.

Cheers,
Kostas

On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:

>
> +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> volunteering as our release managers.
>
> Best Regards,
> Yu
>
>
> On Mon, 26 Oct 2020 at 09:45, Xintong Song <[hidden email]> wrote:
>
> > Thanks Gordan for starting this discussion.
> > My pleasure to help with the release process.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Thanks for starting this discussion Gordon. There are over 100 issues
> > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks
> > > for volunteering as our release managers Gordon and Xintong!
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <[hidden email]
> > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> Xintong and I would like to start a discussion for releasing Flink
> > 1.11.3
> > >> soon.
> > >>
> > >> It seems like we already have a few pressing issues that needs to be
> > >> included in a new hotfix release:
> > >>
> > >> - Heap-based timers’ restore behaviour is causing a critical recovery
> > >> issue for StateFun [1] [2] [3].
> > >> - There are several robustness issues for the FLIP-27 new source API,
> > >> such as [4]. We already have some users using the FLIP-27 API with
> > >> 1.11.x,
> > >> so it would be important to get those fixes in for 1.11.x as well.
> > >>
> > >> Apart from the issues that are already marked as blocker for 1.11.3 in
> > our
> > >> JIRA [5], please let us know in this thread if there is already ongoing
> > >> work for other important fixes that we should try to include.
> > >>
> > >> Xintong and I would like to volunteer for managing this release, and
> > will
> > >> try to communicate the priority of pending blockers over the next few
> > >> days.
> > >> Since the aforementioned issues are quite critical, we’d like to aim
> > >> for a *feature
> > >> freeze by the end of next week (Oct. 30th)* and start the release voting
> > >> process the week after.
> > >> If that is too short of a notice and you might need more time, please
> > let
> > >> us know!
> > >>
> > >> Cheers,
> > >> Gordon
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > >> [5]
> > >>
> > >>
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >>
> > >
> >

Tzu-Li (Gordon) Tai

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Thanks for the replies so far!

Just to provide a brief update on the status of blockers for 1.11.3 so far:

*PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may
return END_OF_INPUT if SplitReader.fetch throws (
https://github.com/apache/flink/pull/13776)

*PR opened, reviewed + close to being merged*
- [FLINK-19741] Timer service should skip restoring from raw keyed stream
if it isn't the writer (https://github.com/apache/flink/pull/13761)
- [FLINK-19748] Raw keyed stream key group iterator should be skipping
unwritten key groups (https://github.com/apache/flink/pull/13772)

*Merged*
- [FLINK-19154] Application mode deletes HA data in case of suspended
ZooKeeper connection
- [FLINK-19569] Upgrade ICU4J to 67.1+

Right now as it seems, progress is mainly blocked on a reviewer for
FLINK-19717.
Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
there are any fixes that should be applied to `release-1.11`.

This is also a reminder, that if there are other blockers that we need to
be aware of, or a need to re-establish estimated time for getting fixes in
and delay the RC for 1.11.3, please do let us know!

Cheers,
Gordon

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability

On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <[hidden email]> wrote:

> +1 for releasing Flink 1.11.3 as it contains a number of important
> fixes and thanks Gordon and Xintong for volunteering.
>
> Cheers,
> Kostas
>
> On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
> >
> > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > volunteering as our release managers.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 26 Oct 2020 at 09:45, Xintong Song <[hidden email]>
> wrote:
> >
> > > Thanks Gordan for starting this discussion.
> > > My pleasure to help with the release process.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <[hidden email]>
> > > wrote:
> > >
> > > > Thanks for starting this discussion Gordon. There are over 100 issues
> > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release.
> Thanks
> > > > for volunteering as our release managers Gordon and Xintong!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> [hidden email]
> > > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Xintong and I would like to start a discussion for releasing Flink
> > > 1.11.3
> > > >> soon.
> > > >>
> > > >> It seems like we already have a few pressing issues that needs to be
> > > >> included in a new hotfix release:
> > > >>
> > > >> - Heap-based timers’ restore behaviour is causing a critical
> recovery
> > > >> issue for StateFun [1] [2] [3].
> > > >> - There are several robustness issues for the FLIP-27 new source
> API,
> > > >> such as [4]. We already have some users using the FLIP-27 API
> with
> > > >> 1.11.x,
> > > >> so it would be important to get those fixes in for 1.11.x as
> well.
> > > >>
> > > >> Apart from the issues that are already marked as blocker for 1.11.3
> in
> > > our
> > > >> JIRA [5], please let us know in this thread if there is already
> ongoing
> > > >> work for other important fixes that we should try to include.
> > > >>
> > > >> Xintong and I would like to volunteer for managing this release, and
> > > will
> > > >> try to communicate the priority of pending blockers over the next
> few
> > > >> days.
> > > >> Since the aforementioned issues are quite critical, we’d like to aim
> > > >> for a *feature
> > > >> freeze by the end of next week (Oct. 30th)* and start the release
> voting
> > > >> process the week after.
> > > >> If that is too short of a notice and you might need more time,
> please
> > > let
> > > >> us know!
> > > >>
> > > >> Cheers,
> > > >> Gordon
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > >> [5]
> > > >>
> > > >>
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >>
> > > >
> > >
>

Steven Wu

Re: [DISCUSS] Releasing Apache Flink 1.11.3

I would love to see this FLIP-27 source interface improvement [1] made to
1.11.3.

[1] https://issues.apache.org/jira/browse/FLINK-19698

On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> Thanks for the replies so far!
>
> Just to provide a brief update on the status of blockers for 1.11.3 so far:
>
>
> *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may
> return END_OF_INPUT if SplitReader.fetch throws (
> https://github.com/apache/flink/pull/13776)
>
> *PR opened, reviewed + close to being merged*
> - [FLINK-19741] Timer service should skip restoring from raw keyed stream
> if it isn't the writer (https://github.com/apache/flink/pull/13761)
> - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> unwritten key groups (https://github.com/apache/flink/pull/13772)
>
> *Merged*
> - [FLINK-19154] Application mode deletes HA data in case of suspended
> ZooKeeper connection
> - [FLINK-19569] Upgrade ICU4J to 67.1+
>
> Right now as it seems, progress is mainly blocked on a reviewer for
> FLINK-19717.
> Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
> there are any fixes that should be applied to `release-1.11`.
>
> This is also a reminder, that if there are other blockers that we need to
> be aware of, or a need to re-establish estimated time for getting fixes in
> and delay the RC for 1.11.3, please do let us know!
>
> Cheers,
> Gordon
>
> [1]
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>
> On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <[hidden email]>
> wrote:
>
> > +1 for releasing Flink 1.11.3 as it contains a number of important
> > fixes and thanks Gordon and Xintong for volunteering.
> >
> > Cheers,
> > Kostas
> >
> > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
> > >
> > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > volunteering as our release managers.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <[hidden email]>
> > wrote:
> > >
> > > > Thanks Gordan for starting this discussion.
> > > > My pleasure to help with the release process.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > Thanks for starting this discussion Gordon. There are over 100
> issues
> > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release.
> > Thanks
> > > > > for volunteering as our release managers Gordon and Xintong!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Xintong and I would like to start a discussion for releasing Flink
> > > > 1.11.3
> > > > >> soon.
> > > > >>
> > > > >> It seems like we already have a few pressing issues that needs to
> be
> > > > >> included in a new hotfix release:
> > > > >>
> > > > >> - Heap-based timers’ restore behaviour is causing a critical
> > recovery
> > > > >> issue for StateFun [1] [2] [3].
> > > > >> - There are several robustness issues for the FLIP-27 new
> source
> > API,
> > > > >> such as [4]. We already have some users using the FLIP-27 API
> > with
> > > > >> 1.11.x,
> > > > >> so it would be important to get those fixes in for 1.11.x as
> > well.
> > > > >>
> > > > >> Apart from the issues that are already marked as blocker for
> 1.11.3
> > in
> > > > our
> > > > >> JIRA [5], please let us know in this thread if there is already
> > ongoing
> > > > >> work for other important fixes that we should try to include.
> > > > >>
> > > > >> Xintong and I would like to volunteer for managing this release,
> and
> > > > will
> > > > >> try to communicate the priority of pending blockers over the next
> > few
> > > > >> days.
> > > > >> Since the aforementioned issues are quite critical, we’d like to
> aim
> > > > >> for a *feature
> > > > >> freeze by the end of next week (Oct. 30th)* and start the release
> > voting
> > > > >> process the week after.
> > > > >> If that is too short of a notice and you might need more time,
> > please
> > > > let
> > > > >> us know!
> > > > >>
> > > > >> Cheers,
> > > > >> Gordon
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > >> [5]
> > > > >>
> > > > >>
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > >>
> > > > >
> > > >
> >
>

Stephan Ewen

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Thanks for starting this.

+1 form my side to backport the FLIP-27 adjustments to 1.11.x.

There were quite a few changes, and I think we need to cherry-pick them all
to not get some inconsistent mix of changes and many merge conflicts.
I made a list below of what we need to add to "release-1.11".

*#### Core Source API Changes to backport (in REVERSE order)*

(Use: "git log
flink-core/src/main/java/org/apache/flink/api/connector/source")

commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
[FLINK-19492][core] Consolidate Source Events between Source API and Split
Reader API

commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
[FLINK-19205][core] Add access to configuration and hostname in the
SourceReaderContext

commit 723e1790744ddc78d19c7c978442af1383f38d33
[hotfix][core] Add to Source Enumerator convenience methods to assign
single split

*#### Connector Base Changes to Backport (in REVERSE order)*

(Use: "git log flink-connectors/flink-connector-base")

commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
[FLINK-19427][FLINK-19489][tests] Fix test conditions for
'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'

commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
[FLINK-19448][connector base] Explicitly check for un-expected condition
that would leave an inconsistent state

commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
[FLINK-19492][core] Consolidate Source Events between Source API and Split
Reader API

commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
[refactor][tests] Move some source test utils from flink-connector-base to
flink-core

commit ee5c4c211c35c70d28252363bbc8400453609977
[FLINK-19251][connectors] Avoid confusing queue handling in
"SplitReader.handleSplitsChanges()"

commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
[FLINK-19250][connectors] Fix error propagation in connector base
(SplitFetcherManager).

commit 8fcca837c55a9216595ee4c03038b52747098dbb
[hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager

commit 4700bb5dde3303cbe98882f6beb7379425717b01
[FLINK-19225][connectors] Various small improvements to SourceReaderBase
(part 2)

commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
[FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
elegantly.

commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
[hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue

commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
[FLINK-19245][connectors] Set default capacity for
FutureCompletingBlockingQueue.

commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
[FLINK-19223][connectors] Simplify Availability Future Model in Base
Connector

commit 511857049ba30c8ff0ee56da551fa4a479dc583e
[FLINK-18128][connectors] Ensure idle split fetchers lead to availability
notifications.

commit a8206467af0830dcb89623ea068b5ca3b3450c92
[refactor][core] Eagerly initialize the FetchTask to support proper unit
testing

commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
[hotfix][tests] Move constants in SplitFetcherTest relevant to only one
test into test method

commit d7625760a75a508bf05bcddc380bb4d62ee1743e
[FLINK-19225][connectors] Various small improvements to SourceReaderBase

commit a5b0d3297748c1be47ad579a88f24df2255a8df1
[FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.

commit f42a3ebc3e81a034b7221a803c153636fef34903
[FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
lightweight.

commit e3d273de822b085183d09b275a445879ff94b350
[FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
support reuse of heavy objects.

commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
[hotfix][testing] Add a set of parameterizable testing mocks for the Split
Reader API

commit 930a07438be1185388d7150640f294dfe2a5d378
[hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'

commit e945ce8a933bc378844782f784ca473c767ca159
[refactor][connectors] Backport of the connector-base exception handling
from the Kafka Connector Pull Request

On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]> wrote:

> I would love to see this FLIP-27 source interface improvement [1] made to
> 1.11.3.
>
> [1] https://issues.apache.org/jira/browse/FLINK-19698
>
> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> > Thanks for the replies so far!
> >
> > Just to provide a brief update on the status of blockers for 1.11.3 so
> far:
> >
> >
> > *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext
> may
> > return END_OF_INPUT if SplitReader.fetch throws (
> > https://github.com/apache/flink/pull/13776)
> >
> > *PR opened, reviewed + close to being merged*
> > - [FLINK-19741] Timer service should skip restoring from raw keyed stream
> > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> > unwritten key groups (https://github.com/apache/flink/pull/13772)
> >
> > *Merged*
> > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > ZooKeeper connection
> > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >
> > Right now as it seems, progress is mainly blocked on a reviewer for
> > FLINK-19717.
> > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
> > there are any fixes that should be applied to `release-1.11`.
> >
> > This is also a reminder, that if there are other blockers that we need to
> > be aware of, or a need to re-establish estimated time for getting fixes
> in
> > and delay the RC for 1.11.3, please do let us know!
> >
> > Cheers,
> > Gordon
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >
> > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <[hidden email]>
> > wrote:
> >
> > > +1 for releasing Flink 1.11.3 as it contains a number of important
> > > fixes and thanks Gordon and Xintong for volunteering.
> > >
> > > Cheers,
> > > Kostas
> > >
> > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
> > > >
> > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > > volunteering as our release managers.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <[hidden email]>
> > > wrote:
> > > >
> > > > > Thanks Gordan for starting this discussion.
> > > > > My pleasure to help with the release process.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for starting this discussion Gordon. There are over 100
> > issues
> > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> release.
> > > Thanks
> > > > > > for volunteering as our release managers Gordon and Xintong!
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> Xintong and I would like to start a discussion for releasing
> Flink
> > > > > 1.11.3
> > > > > >> soon.
> > > > > >>
> > > > > >> It seems like we already have a few pressing issues that needs
> to
> > be
> > > > > >> included in a new hotfix release:
> > > > > >>
> > > > > >> - Heap-based timers’ restore behaviour is causing a critical
> > > recovery
> > > > > >> issue for StateFun [1] [2] [3].
> > > > > >> - There are several robustness issues for the FLIP-27 new
> > source
> > > API,
> > > > > >> such as [4]. We already have some users using the FLIP-27 API
> > > with
> > > > > >> 1.11.x,
> > > > > >> so it would be important to get those fixes in for 1.11.x as
> > > well.
> > > > > >>
> > > > > >> Apart from the issues that are already marked as blocker for
> > 1.11.3
> > > in
> > > > > our
> > > > > >> JIRA [5], please let us know in this thread if there is already
> > > ongoing
> > > > > >> work for other important fixes that we should try to include.
> > > > > >>
> > > > > >> Xintong and I would like to volunteer for managing this release,
> > and
> > > > > will
> > > > > >> try to communicate the priority of pending blockers over the
> next
> > > few
> > > > > >> days.
> > > > > >> Since the aforementioned issues are quite critical, we’d like to
> > aim
> > > > > >> for a *feature
> > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> release
> > > voting
> > > > > >> process the week after.
> > > > > >> If that is too short of a notice and you might need more time,
> > > please
> > > > > let
> > > > > >> us know!
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Gordon
> > > > > >>
> > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > >> [5]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>

Jingsong Li

Re: [DISCUSS] Releasing Apache Flink 1.11.3

+1 to backport the FLIP-27 adjustments to 1.11.x.

If possible, that would be great. Many people are looking forward to the
FLIP-27 interface, but they don't want to take the risk to upgrade to 1.12
(And wait 1.12). After all, 1.11 is a relatively stable version.

Best,
Jingsong

On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]> wrote:

> Thanks for starting this.
>
> +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>
> There were quite a few changes, and I think we need to cherry-pick them all
> to not get some inconsistent mix of changes and many merge conflicts.
> I made a list below of what we need to add to "release-1.11".
>
> *#### Core Source API Changes to backport (in REVERSE order)*
>
> (Use: "git log
> flink-core/src/main/java/org/apache/flink/api/connector/source")
>
> commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> [FLINK-19492][core] Consolidate Source Events between Source API and Split
> Reader API
>
> commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> [FLINK-19205][core] Add access to configuration and hostname in the
> SourceReaderContext
>
> commit 723e1790744ddc78d19c7c978442af1383f38d33
> [hotfix][core] Add to Source Enumerator convenience methods to assign
> single split
>
> *#### Connector Base Changes to Backport (in REVERSE order)*
>
> (Use: "git log flink-connectors/flink-connector-base")
>
> commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>
> commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> [FLINK-19448][connector base] Explicitly check for un-expected condition
> that would leave an inconsistent state
>
> commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> [FLINK-19492][core] Consolidate Source Events between Source API and Split
> Reader API
>
> commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> [refactor][tests] Move some source test utils from flink-connector-base to
> flink-core
>
> commit ee5c4c211c35c70d28252363bbc8400453609977
> [FLINK-19251][connectors] Avoid confusing queue handling in
> "SplitReader.handleSplitsChanges()"
>
> commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> [FLINK-19250][connectors] Fix error propagation in connector base
> (SplitFetcherManager).
>
> commit 8fcca837c55a9216595ee4c03038b52747098dbb
> [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
>
> commit 4700bb5dde3303cbe98882f6beb7379425717b01
> [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> (part 2)
>
> commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> elegantly.
>
> commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
>
> commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> [FLINK-19245][connectors] Set default capacity for
> FutureCompletingBlockingQueue.
>
> commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> [FLINK-19223][connectors] Simplify Availability Future Model in Base
> Connector
>
> commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> [FLINK-18128][connectors] Ensure idle split fetchers lead to availability
> notifications.
>
> commit a8206467af0830dcb89623ea068b5ca3b3450c92
> [refactor][core] Eagerly initialize the FetchTask to support proper unit
> testing
>
> commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> test into test method
>
> commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> [FLINK-19225][connectors] Various small improvements to SourceReaderBase
>
> commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>
> commit f42a3ebc3e81a034b7221a803c153636fef34903
> [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> lightweight.
>
> commit e3d273de822b085183d09b275a445879ff94b350
> [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> support reuse of heavy objects.
>
> commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> [hotfix][testing] Add a set of parameterizable testing mocks for the Split
> Reader API
>
> commit 930a07438be1185388d7150640f294dfe2a5d378
> [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
>
> commit e945ce8a933bc378844782f784ca473c767ca159
> [refactor][connectors] Backport of the connector-base exception handling
> from the Kafka Connector Pull Request
>
>
> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]> wrote:
>
> > I would love to see this FLIP-27 source interface improvement [1] made to
> > 1.11.3.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >
> > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> [hidden email]>
> > wrote:
> >
> > > Thanks for the replies so far!
> > >
> > > Just to provide a brief update on the status of blockers for 1.11.3 so
> > far:
> > >
> > >
> > > *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext
> > may
> > > return END_OF_INPUT if SplitReader.fetch throws (
> > > https://github.com/apache/flink/pull/13776)
> > >
> > > *PR opened, reviewed + close to being merged*
> > > - [FLINK-19741] Timer service should skip restoring from raw keyed
> stream
> > > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > > - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> > > unwritten key groups (https://github.com/apache/flink/pull/13772)
> > >
> > > *Merged*
> > > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > > ZooKeeper connection
> > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >
> > > Right now as it seems, progress is mainly blocked on a reviewer for
> > > FLINK-19717.
> > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see
> if
> > > there are any fixes that should be applied to `release-1.11`.
> > >
> > > This is also a reminder, that if there are other blockers that we need
> to
> > > be aware of, or a need to re-establish estimated time for getting fixes
> > in
> > > and delay the RC for 1.11.3, please do let us know!
> > >
> > > Cheers,
> > > Gordon
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >
> > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <[hidden email]>
> > > wrote:
> > >
> > > > +1 for releasing Flink 1.11.3 as it contains a number of important
> > > > fixes and thanks Gordon and Xintong for volunteering.
> > > >
> > > > Cheers,
> > > > Kostas
> > > >
> > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
> > > > >
> > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > > > volunteering as our release managers.
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Thanks Gordan for starting this discussion.
> > > > > > My pleasure to help with the release process.
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for starting this discussion Gordon. There are over 100
> > > issues
> > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> > release.
> > > > Thanks
> > > > > > > for volunteering as our release managers Gordon and Xintong!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> Xintong and I would like to start a discussion for releasing
> > Flink
> > > > > > 1.11.3
> > > > > > >> soon.
> > > > > > >>
> > > > > > >> It seems like we already have a few pressing issues that needs
> > to
> > > be
> > > > > > >> included in a new hotfix release:
> > > > > > >>
> > > > > > >> - Heap-based timers’ restore behaviour is causing a
> critical
> > > > recovery
> > > > > > >> issue for StateFun [1] [2] [3].
> > > > > > >> - There are several robustness issues for the FLIP-27 new
> > > source
> > > > API,
> > > > > > >> such as [4]. We already have some users using the FLIP-27
> API
> > > > with
> > > > > > >> 1.11.x,
> > > > > > >> so it would be important to get those fixes in for 1.11.x
> as
> > > > well.
> > > > > > >>
> > > > > > >> Apart from the issues that are already marked as blocker for
> > > 1.11.3
> > > > in
> > > > > > our
> > > > > > >> JIRA [5], please let us know in this thread if there is
> already
> > > > ongoing
> > > > > > >> work for other important fixes that we should try to include.
> > > > > > >>
> > > > > > >> Xintong and I would like to volunteer for managing this
> release,
> > > and
> > > > > > will
> > > > > > >> try to communicate the priority of pending blockers over the
> > next
> > > > few
> > > > > > >> days.
> > > > > > >> Since the aforementioned issues are quite critical, we’d like
> to
> > > aim
> > > > > > >> for a *feature
> > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> > release
> > > > voting
> > > > > > >> process the week after.
> > > > > > >> If that is too short of a notice and you might need more time,
> > > > please
> > > > > > let
> > > > > > >> us know!
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> Gordon
> > > > > > >>
> > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > >> [5]
> > > > > > >>
> > > > > > >>
> > > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

--
Best, Jingsong Lee

Xintong Song

Re: [DISCUSS] Releasing Apache Flink 1.11.3

> +1 to backport the FLIP-27 adjustments to 1.11.x.
>
> If possible, that would be great. Many people are looking forward to the
> FLIP-27 interface, but they don't want to take the risk to upgrade to 1.12
> (And wait 1.12). After all, 1.11 is a relatively stable version.
>
> Best,
> Jingsong
>
> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]> wrote:
>
> > Thanks for starting this.
> >
> > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> >
> > There were quite a few changes, and I think we need to cherry-pick them
> all
> > to not get some inconsistent mix of changes and many merge conflicts.
> > I made a list below of what we need to add to "release-1.11".
> >
> > *#### Core Source API Changes to backport (in REVERSE order)*
> >
> > (Use: "git log
> > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >
> > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > [FLINK-19492][core] Consolidate Source Events between Source API and
> Split
> > Reader API
> >
> > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > [FLINK-19205][core] Add access to configuration and hostname in the
> > SourceReaderContext
> >
> > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > [hotfix][core] Add to Source Enumerator convenience methods to assign
> > single split
> >
> > *#### Connector Base Changes to Backport (in REVERSE order)*
> >
> > (Use: "git log flink-connectors/flink-connector-base")
> >
> > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >
> > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > [FLINK-19448][connector base] Explicitly check for un-expected condition
> > that would leave an inconsistent state
> >
> > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > [FLINK-19492][core] Consolidate Source Events between Source API and
> Split
> > Reader API
> >
> > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > [refactor][tests] Move some source test utils from flink-connector-base
> to
> > flink-core
> >
> > commit ee5c4c211c35c70d28252363bbc8400453609977
> > [FLINK-19251][connectors] Avoid confusing queue handling in
> > "SplitReader.handleSplitsChanges()"
> >
> > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > [FLINK-19250][connectors] Fix error propagation in connector base
> > (SplitFetcherManager).
> >
> > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
> >
> > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> > (part 2)
> >
> > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> > elegantly.
> >
> > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
> >
> > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > [FLINK-19245][connectors] Set default capacity for
> > FutureCompletingBlockingQueue.
> >
> > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > [FLINK-19223][connectors] Simplify Availability Future Model in Base
> > Connector
> >
> > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > [FLINK-18128][connectors] Ensure idle split fetchers lead to availability
> > notifications.
> >
> > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > [refactor][core] Eagerly initialize the FetchTask to support proper unit
> > testing
> >
> > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> > test into test method
> >
> > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> >
> > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
> >
> > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> > lightweight.
> >
> > commit e3d273de822b085183d09b275a445879ff94b350
> > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> > support reuse of heavy objects.
> >
> > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > [hotfix][testing] Add a set of parameterizable testing mocks for the
> Split
> > Reader API
> >
> > commit 930a07438be1185388d7150640f294dfe2a5d378
> > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
> >
> > commit e945ce8a933bc378844782f784ca473c767ca159
> > [refactor][connectors] Backport of the connector-base exception handling
> > from the Kafka Connector Pull Request
> >
> >
> > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]> wrote:
> >
> > > I would love to see this FLIP-27 source interface improvement [1] made
> to
> > > 1.11.3.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > >
> > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > [hidden email]>
> > > wrote:
> > >
> > > > Thanks for the replies so far!
> > > >
> > > > Just to provide a brief update on the status of blockers for 1.11.3
> so
> > > far:
> > > >
> > > >
> > > > *PR opened, pending reviewer*- [FLINK-19717]
> SourceReaderBase.pollNext
> > > may
> > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > https://github.com/apache/flink/pull/13776)
> > > >
> > > > *PR opened, reviewed + close to being merged*
> > > > - [FLINK-19741] Timer service should skip restoring from raw keyed
> > stream
> > > > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> skipping
> > > > unwritten key groups (https://github.com/apache/flink/pull/13772)
> > > >
> > > > *Merged*
> > > > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > > > ZooKeeper connection
> > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >
> > > > Right now as it seems, progress is mainly blocked on a reviewer for
> > > > FLINK-19717.
> > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see
> > if
> > > > there are any fixes that should be applied to `release-1.11`.
> > > >
> > > > This is also a reminder, that if there are other blockers that we
> need
> > to
> > > > be aware of, or a need to re-establish estimated time for getting
> fixes
> > > in
> > > > and delay the RC for 1.11.3, please do let us know!
> > > >
> > > > Cheers,
> > > > Gordon
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >
> > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <[hidden email]>
> > > > wrote:
> > > >
> > > > > +1 for releasing Flink 1.11.3 as it contains a number of important
> > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > > >
> > > > > Cheers,
> > > > > Kostas
> > > > >
> > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
> > > > > >
> > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > > > > volunteering as our release managers.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Gordan for starting this discussion.
> > > > > > > My pleasure to help with the release process.
> > > > > > >
> > > > > > > Thank you~
> > > > > > >
> > > > > > > Xintong Song
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for starting this discussion Gordon. There are over
> 100
> > > > issues
> > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> > > release.
> > > > > Thanks
> > > > > > > > for volunteering as our release managers Gordon and Xintong!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> Xintong and I would like to start a discussion for releasing
> > > Flink
> > > > > > > 1.11.3
> > > > > > > >> soon.
> > > > > > > >>
> > > > > > > >> It seems like we already have a few pressing issues that
> needs
> > > to
> > > > be
> > > > > > > >> included in a new hotfix release:
> > > > > > > >>
> > > > > > > >> - Heap-based timers’ restore behaviour is causing a
> > critical
> > > > > recovery
> > > > > > > >> issue for StateFun [1] [2] [3].
> > > > > > > >> - There are several robustness issues for the FLIP-27 new
> > > > source
> > > > > API,
> > > > > > > >> such as [4]. We already have some users using the FLIP-27
> > API
> > > > > with
> > > > > > > >> 1.11.x,
> > > > > > > >> so it would be important to get those fixes in for 1.11.x
> > as
> > > > > well.
> > > > > > > >>
> > > > > > > >> Apart from the issues that are already marked as blocker for
> > > > 1.11.3
> > > > > in
> > > > > > > our
> > > > > > > >> JIRA [5], please let us know in this thread if there is
> > already
> > > > > ongoing
> > > > > > > >> work for other important fixes that we should try to
> include.
> > > > > > > >>
> > > > > > > >> Xintong and I would like to volunteer for managing this
> > release,
> > > > and
> > > > > > > will
> > > > > > > >> try to communicate the priority of pending blockers over the
> > > next
> > > > > few
> > > > > > > >> days.
> > > > > > > >> Since the aforementioned issues are quite critical, we’d
> like
> > to
> > > > aim
> > > > > > > >> for a *feature
> > > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> > > release
> > > > > voting
> > > > > > > >> process the week after.
> > > > > > > >> If that is too short of a notice and you might need more
> time,
> > > > > please
> > > > > > > let
> > > > > > > >> us know!
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> Gordon
> > > > > > > >>
> > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > > >> [5]
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

Tzu-Li (Gordon) Tai

Re: [DISCUSS] Releasing Apache Flink 1.11.3

@Stephan Ewen <[hidden email]>
Are there already plans or ongoing efforts for backporting the list of
FLIP-27 changes that you posted?

On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <[hidden email]> wrote:

> Hi folks,
>
> Just to provide some updates concerning the status on the
> test instabilities.
>
> Currently, we have 30 unresolved tickets labeled with `Affects Version`
> 1.11.x.
>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>
> Among the 30 tickets, 11 of them are:
> - Have occured in the recent 3 months
> - Not confirmed to be pure testability issues
> - Not confirmed to be rare condition cases
>
> It would be nice if someone familiar with these components can take a look
> into these issues.
>
> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
> - https://issues.apache.org/jira/browse/FLINK-19775
> (SystemProcessingTimeServiceTest)
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <[hidden email]>
> wrote:
>
> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >
> > If possible, that would be great. Many people are looking forward to the
> > FLIP-27 interface, but they don't want to take the risk to upgrade to
> 1.12
> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]> wrote:
> >
> > > Thanks for starting this.
> > >
> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> > >
> > > There were quite a few changes, and I think we need to cherry-pick them
> > all
> > > to not get some inconsistent mix of changes and many merge conflicts.
> > > I made a list below of what we need to add to "release-1.11".
> > >
> > > *#### Core Source API Changes to backport (in REVERSE order)*
> > >
> > > (Use: "git log
> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >
> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> > Split
> > > Reader API
> > >
> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > [FLINK-19205][core] Add access to configuration and hostname in the
> > > SourceReaderContext
> > >
> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > [hotfix][core] Add to Source Enumerator convenience methods to assign
> > > single split
> > >
> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> > >
> > > (Use: "git log flink-connectors/flink-connector-base")
> > >
> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >
> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > [FLINK-19448][connector base] Explicitly check for un-expected
> condition
> > > that would leave an inconsistent state
> > >
> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> > Split
> > > Reader API
> > >
> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > [refactor][tests] Move some source test utils from flink-connector-base
> > to
> > > flink-core
> > >
> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > > "SplitReader.handleSplitsChanges()"
> > >
> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > [FLINK-19250][connectors] Fix error propagation in connector base
> > > (SplitFetcherManager).
> > >
> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
> > >
> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > [FLINK-19225][connectors] Various small improvements to
> SourceReaderBase
> > > (part 2)
> > >
> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> > > elegantly.
> > >
> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
> > >
> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > [FLINK-19245][connectors] Set default capacity for
> > > FutureCompletingBlockingQueue.
> > >
> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > [FLINK-19223][connectors] Simplify Availability Future Model in Base
> > > Connector
> > >
> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> availability
> > > notifications.
> > >
> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > [refactor][core] Eagerly initialize the FetchTask to support proper
> unit
> > > testing
> > >
> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> > > test into test method
> > >
> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > [FLINK-19225][connectors] Various small improvements to
> SourceReaderBase
> > >
> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
> > >
> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> > > lightweight.
> > >
> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> > > support reuse of heavy objects.
> > >
> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > [hotfix][testing] Add a set of parameterizable testing mocks for the
> > Split
> > > Reader API
> > >
> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
> > >
> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > [refactor][connectors] Backport of the connector-base exception
> handling
> > > from the Kafka Connector Pull Request
> > >
> > >
> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]>
> wrote:
> > >
> > > > I would love to see this FLIP-27 source interface improvement [1]
> made
> > to
> > > > 1.11.3.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > >
> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Thanks for the replies so far!
> > > > >
> > > > > Just to provide a brief update on the status of blockers for 1.11.3
> > so
> > > > far:
> > > > >
> > > > >
> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > SourceReaderBase.pollNext
> > > > may
> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > > https://github.com/apache/flink/pull/13776)
> > > > >
> > > > > *PR opened, reviewed + close to being merged*
> > > > > - [FLINK-19741] Timer service should skip restoring from raw keyed
> > > stream
> > > > > if it isn't the writer (https://github.com/apache/flink/pull/13761
> )
> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> > skipping
> > > > > unwritten key groups (https://github.com/apache/flink/pull/13772)
> > > > >
> > > > > *Merged*
> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> suspended
> > > > > ZooKeeper connection
> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > >
> > > > > Right now as it seems, progress is mainly blocked on a reviewer for
> > > > > FLINK-19717.
> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to
> see
> > > if
> > > > > there are any fixes that should be applied to `release-1.11`.
> > > > >
> > > > > This is also a reminder, that if there are other blockers that we
> > need
> > > to
> > > > > be aware of, or a need to re-establish estimated time for getting
> > fixes
> > > > in
> > > > > and delay the RC for 1.11.3, please do let us know!
> > > > >
> > > > > Cheers,
> > > > > Gordon
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > >
> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> important
> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > > > >
> > > > > > Cheers,
> > > > > > Kostas
> > > > > >
> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
> > > > > > >
> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong
> for
> > > > > > > volunteering as our release managers.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Yu
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > [hidden email]>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Gordan for starting this discussion.
> > > > > > > > My pleasure to help with the release process.
> > > > > > > >
> > > > > > > > Thank you~
> > > > > > > >
> > > > > > > > Xintong Song
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for starting this discussion Gordon. There are over
> > 100
> > > > > issues
> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> > > > release.
> > > > > > Thanks
> > > > > > > > > for volunteering as our release managers Gordon and
> Xintong!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Till
> > > > > > > > >
> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> Xintong and I would like to start a discussion for
> releasing
> > > > Flink
> > > > > > > > 1.11.3
> > > > > > > > >> soon.
> > > > > > > > >>
> > > > > > > > >> It seems like we already have a few pressing issues that
> > needs
> > > > to
> > > > > be
> > > > > > > > >> included in a new hotfix release:
> > > > > > > > >>
> > > > > > > > >> - Heap-based timers’ restore behaviour is causing a
> > > critical
> > > > > > recovery
> > > > > > > > >> issue for StateFun [1] [2] [3].
> > > > > > > > >> - There are several robustness issues for the FLIP-27
> new
> > > > > source
> > > > > > API,
> > > > > > > > >> such as [4]. We already have some users using the
> FLIP-27
> > > API
> > > > > > with
> > > > > > > > >> 1.11.x,
> > > > > > > > >> so it would be important to get those fixes in for
> 1.11.x
> > > as
> > > > > > well.
> > > > > > > > >>
> > > > > > > > >> Apart from the issues that are already marked as blocker
> for
> > > > > 1.11.3
> > > > > > in
> > > > > > > > our
> > > > > > > > >> JIRA [5], please let us know in this thread if there is
> > > already
> > > > > > ongoing
> > > > > > > > >> work for other important fixes that we should try to
> > include.
> > > > > > > > >>
> > > > > > > > >> Xintong and I would like to volunteer for managing this
> > > release,
> > > > > and
> > > > > > > > will
> > > > > > > > >> try to communicate the priority of pending blockers over
> the
> > > > next
> > > > > > few
> > > > > > > > >> days.
> > > > > > > > >> Since the aforementioned issues are quite critical, we’d
> > like
> > > to
> > > > > aim
> > > > > > > > >> for a *feature
> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> > > > release
> > > > > > voting
> > > > > > > > >> process the week after.
> > > > > > > > >> If that is too short of a notice and you might need more
> > time,
> > > > > > please
> > > > > > > > let
> > > > > > > > >> us know!
> > > > > > > > >>
> > > > > > > > >> Cheers,
> > > > > > > > >> Gordon
> > > > > > > > >>
> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > > > >> [5]
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

Stephan Ewen

Re: [DISCUSS] Releasing Apache Flink 1.11.3

I have started with backporting the source API changes. Some minor
conflicts to solve, will need a bit more to finish this.

On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> @Stephan Ewen <[hidden email]>
> Are there already plans or ongoing efforts for backporting the list of
> FLIP-27 changes that you posted?
>
> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <[hidden email]>
> wrote:
>
>> Hi folks,
>>
>> Just to provide some updates concerning the status on the
>> test instabilities.
>>
>> Currently, we have 30 unresolved tickets labeled with `Affects Version`
>> 1.11.x.
>>
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>>
>> Among the 30 tickets, 11 of them are:
>> - Have occured in the recent 3 months
>> - Not confirmed to be pure testability issues
>> - Not confirmed to be rare condition cases
>>
>> It would be nice if someone familiar with these components can take a look
>> into these issues.
>>
>> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
>> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
>> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
>> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
>> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
>> - https://issues.apache.org/jira/browse/FLINK-19775
>> (SystemProcessingTimeServiceTest)
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <[hidden email]>
>> wrote:
>>
>> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> >
>> > If possible, that would be great. Many people are looking forward to the
>> > FLIP-27 interface, but they don't want to take the risk to upgrade to
>> 1.12
>> > (And wait 1.12). After all, 1.11 is a relatively stable version.
>> >
>> > Best,
>> > Jingsong
>> >
>> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]> wrote:
>> >
>> > > Thanks for starting this.
>> > >
>> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>> > >
>> > > There were quite a few changes, and I think we need to cherry-pick
>> them
>> > all
>> > > to not get some inconsistent mix of changes and many merge conflicts.
>> > > I made a list below of what we need to add to "release-1.11".
>> > >
>> > > *#### Core Source API Changes to backport (in REVERSE order)*
>> > >
>> > > (Use: "git log
>> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > >
>> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > [FLINK-19492][core] Consolidate Source Events between Source API and
>> > Split
>> > > Reader API
>> > >
>> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > > [FLINK-19205][core] Add access to configuration and hostname in the
>> > > SourceReaderContext
>> > >
>> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> > > [hotfix][core] Add to Source Enumerator convenience methods to assign
>> > > single split
>> > >
>> > > *#### Connector Base Changes to Backport (in REVERSE order)*
>> > >
>> > > (Use: "git log flink-connectors/flink-connector-base")
>> > >
>> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
>> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > >
>> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > > [FLINK-19448][connector base] Explicitly check for un-expected
>> condition
>> > > that would leave an inconsistent state
>> > >
>> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > [FLINK-19492][core] Consolidate Source Events between Source API and
>> > Split
>> > > Reader API
>> > >
>> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > > [refactor][tests] Move some source test utils from
>> flink-connector-base
>> > to
>> > > flink-core
>> > >
>> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> > > [FLINK-19251][connectors] Avoid confusing queue handling in
>> > > "SplitReader.handleSplitsChanges()"
>> > >
>> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > > [FLINK-19250][connectors] Fix error propagation in connector base
>> > > (SplitFetcherManager).
>> > >
>> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> > > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
>> > >
>> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> > > [FLINK-19225][connectors] Various small improvements to
>> SourceReaderBase
>> > > (part 2)
>> > >
>> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
>> > > elegantly.
>> > >
>> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > > [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
>> > >
>> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > > [FLINK-19245][connectors] Set default capacity for
>> > > FutureCompletingBlockingQueue.
>> > >
>> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > > [FLINK-19223][connectors] Simplify Availability Future Model in Base
>> > > Connector
>> > >
>> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
>> availability
>> > > notifications.
>> > >
>> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> > > [refactor][core] Eagerly initialize the FetchTask to support proper
>> unit
>> > > testing
>> > >
>> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only
>> one
>> > > test into test method
>> > >
>> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > > [FLINK-19225][connectors] Various small improvements to
>> SourceReaderBase
>> > >
>> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>> > >
>> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
>> > > lightweight.
>> > >
>> > > commit e3d273de822b085183d09b275a445879ff94b350
>> > > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds
>> to
>> > > support reuse of heavy objects.
>> > >
>> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > > [hotfix][testing] Add a set of parameterizable testing mocks for the
>> > Split
>> > > Reader API
>> > >
>> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> > > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
>> > >
>> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> > > [refactor][connectors] Backport of the connector-base exception
>> handling
>> > > from the Kafka Connector Pull Request
>> > >
>> > >
>> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]>
>> wrote:
>> > >
>> > > > I would love to see this FLIP-27 source interface improvement [1]
>> made
>> > to
>> > > > 1.11.3.
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
>> > > >
>> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
>> > > [hidden email]>
>> > > > wrote:
>> > > >
>> > > > > Thanks for the replies so far!
>> > > > >
>> > > > > Just to provide a brief update on the status of blockers for
>> 1.11.3
>> > so
>> > > > far:
>> > > > >
>> > > > >
>> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> > SourceReaderBase.pollNext
>> > > > may
>> > > > > return END_OF_INPUT if SplitReader.fetch throws (
>> > > > > https://github.com/apache/flink/pull/13776)
>> > > > >
>> > > > > *PR opened, reviewed + close to being merged*
>> > > > > - [FLINK-19741] Timer service should skip restoring from raw keyed
>> > > stream
>> > > > > if it isn't the writer (
>> https://github.com/apache/flink/pull/13761)
>> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
>> > skipping
>> > > > > unwritten key groups (https://github.com/apache/flink/pull/13772)
>> > > > >
>> > > > > *Merged*
>> > > > > - [FLINK-19154] Application mode deletes HA data in case of
>> suspended
>> > > > > ZooKeeper connection
>> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > > > >
>> > > > > Right now as it seems, progress is mainly blocked on a reviewer
>> for
>> > > > > FLINK-19717.
>> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to
>> see
>> > > if
>> > > > > there are any fixes that should be applied to `release-1.11`.
>> > > > >
>> > > > > This is also a reminder, that if there are other blockers that we
>> > need
>> > > to
>> > > > > be aware of, or a need to re-establish estimated time for getting
>> > fixes
>> > > > in
>> > > > > and delay the RC for 1.11.3, please do let us know!
>> > > > >
>> > > > > Cheers,
>> > > > > Gordon
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > > > >
>> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
>> [hidden email]>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
>> important
>> > > > > > fixes and thanks Gordon and Xintong for volunteering.
>> > > > > >
>> > > > > > Cheers,
>> > > > > > Kostas
>> > > > > >
>> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]> wrote:
>> > > > > > >
>> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong
>> for
>> > > > > > > volunteering as our release managers.
>> > > > > > >
>> > > > > > > Best Regards,
>> > > > > > > Yu
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
>> > [hidden email]>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Thanks Gordan for starting this discussion.
>> > > > > > > > My pleasure to help with the release process.
>> > > > > > > >
>> > > > > > > > Thank you~
>> > > > > > > >
>> > > > > > > > Xintong Song
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
>> > > > [hidden email]
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks for starting this discussion Gordon. There are over
>> > 100
>> > > > > issues
>> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
>> > > > release.
>> > > > > > Thanks
>> > > > > > > > > for volunteering as our release managers Gordon and
>> Xintong!
>> > > > > > > > >
>> > > > > > > > > Cheers,
>> > > > > > > > > Till
>> > > > > > > > >
>> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
>> > > > > > [hidden email]
>> > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi,
>> > > > > > > > >>
>> > > > > > > > >> Xintong and I would like to start a discussion for
>> releasing
>> > > > Flink
>> > > > > > > > 1.11.3
>> > > > > > > > >> soon.
>> > > > > > > > >>
>> > > > > > > > >> It seems like we already have a few pressing issues that
>> > needs
>> > > > to
>> > > > > be
>> > > > > > > > >> included in a new hotfix release:
>> > > > > > > > >>
>> > > > > > > > >> - Heap-based timers’ restore behaviour is causing a
>> > > critical
>> > > > > > recovery
>> > > > > > > > >> issue for StateFun [1] [2] [3].
>> > > > > > > > >> - There are several robustness issues for the FLIP-27
>> new
>> > > > > source
>> > > > > > API,
>> > > > > > > > >> such as [4]. We already have some users using the
>> FLIP-27
>> > > API
>> > > > > > with
>> > > > > > > > >> 1.11.x,
>> > > > > > > > >> so it would be important to get those fixes in for
>> 1.11.x
>> > > as
>> > > > > > well.
>> > > > > > > > >>
>> > > > > > > > >> Apart from the issues that are already marked as blocker
>> for
>> > > > > 1.11.3
>> > > > > > in
>> > > > > > > > our
>> > > > > > > > >> JIRA [5], please let us know in this thread if there is
>> > > already
>> > > > > > ongoing
>> > > > > > > > >> work for other important fixes that we should try to
>> > include.
>> > > > > > > > >>
>> > > > > > > > >> Xintong and I would like to volunteer for managing this
>> > > release,
>> > > > > and
>> > > > > > > > will
>> > > > > > > > >> try to communicate the priority of pending blockers over
>> the
>> > > > next
>> > > > > > few
>> > > > > > > > >> days.
>> > > > > > > > >> Since the aforementioned issues are quite critical, we’d
>> > like
>> > > to
>> > > > > aim
>> > > > > > > > >> for a *feature
>> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
>> > > > release
>> > > > > > voting
>> > > > > > > > >> process the week after.
>> > > > > > > > >> If that is too short of a notice and you might need more
>> > time,
>> > > > > > please
>> > > > > > > > let
>> > > > > > > > >> us know!
>> > > > > > > > >>
>> > > > > > > > >> Cheers,
>> > > > > > > > >> Gordon
>> > > > > > > > >>
>> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
>> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
>> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
>> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
>> > > > > > > > >> [5]
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> > --
>> > Best, Jingsong Lee
>> >
>>
>

Steven Wu

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Stephan, it will be great if we can also backport the DataStreamUtils
related commits that help with collecting output from unbounded streams.
e.g.

https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc

I tried to copy and paste the code to unblock myself. but it quickly got
into the rabbit hole of more and more code.

On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]> wrote:

> I have started with backporting the source API changes. Some minor
> conflicts to solve, will need a bit more to finish this.
>
> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> > @Stephan Ewen <[hidden email]>
> > Are there already plans or ongoing efforts for backporting the list of
> > FLIP-27 changes that you posted?
> >
> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <[hidden email]>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Just to provide some updates concerning the status on the
> >> test instabilities.
> >>
> >> Currently, we have 30 unresolved tickets labeled with `Affects Version`
> >> 1.11.x.
> >>
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >>
> >> Among the 30 tickets, 11 of them are:
> >> - Have occured in the recent 3 months
> >> - Not confirmed to be pure testability issues
> >> - Not confirmed to be rare condition cases
> >>
> >> It would be nice if someone familiar with these components can take a
> look
> >> into these issues.
> >>
> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
> >> - https://issues.apache.org/jira/browse/FLINK-19775
> >> (SystemProcessingTimeServiceTest)
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <[hidden email]>
> >> wrote:
> >>
> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >> >
> >> > If possible, that would be great. Many people are looking forward to
> the
> >> > FLIP-27 interface, but they don't want to take the risk to upgrade to
> >> 1.12
> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> >> >
> >> > Best,
> >> > Jingsong
> >> >
> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]>
> wrote:
> >> >
> >> > > Thanks for starting this.
> >> > >
> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> >> > >
> >> > > There were quite a few changes, and I think we need to cherry-pick
> >> them
> >> > all
> >> > > to not get some inconsistent mix of changes and many merge
> conflicts.
> >> > > I made a list below of what we need to add to "release-1.11".
> >> > >
> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> >> > >
> >> > > (Use: "git log
> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >> > >
> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> >> > Split
> >> > > Reader API
> >> > >
> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >> > > [FLINK-19205][core] Add access to configuration and hostname in the
> >> > > SourceReaderContext
> >> > >
> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
> assign
> >> > > single split
> >> > >
> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> >> > >
> >> > > (Use: "git log flink-connectors/flink-connector-base")
> >> > >
> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >> > >
> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
> >> condition
> >> > > that would leave an inconsistent state
> >> > >
> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> >> > Split
> >> > > Reader API
> >> > >
> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >> > > [refactor][tests] Move some source test utils from
> >> flink-connector-base
> >> > to
> >> > > flink-core
> >> > >
> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> >> > > "SplitReader.handleSplitsChanges()"
> >> > >
> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >> > > [FLINK-19250][connectors] Fix error propagation in connector base
> >> > > (SplitFetcherManager).
> >> > >
> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> >> > > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
> >> > >
> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> >> > > [FLINK-19225][connectors] Various small improvements to
> >> SourceReaderBase
> >> > > (part 2)
> >> > >
> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> >> > > elegantly.
> >> > >
> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >> > > [hotfix][tests] Extend test coverage for
> FutureCompletingBlockingQueue
> >> > >
> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >> > > [FLINK-19245][connectors] Set default capacity for
> >> > > FutureCompletingBlockingQueue.
> >> > >
> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in Base
> >> > > Connector
> >> > >
> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> >> availability
> >> > > notifications.
> >> > >
> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> >> > > [refactor][core] Eagerly initialize the FetchTask to support proper
> >> unit
> >> > > testing
> >> > >
> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only
> >> one
> >> > > test into test method
> >> > >
> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> >> > > [FLINK-19225][connectors] Various small improvements to
> >> SourceReaderBase
> >> > >
> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
> >> > >
> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds
> more
> >> > > lightweight.
> >> > >
> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> >> > > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds
> >> to
> >> > > support reuse of heavy objects.
> >> > >
> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for the
> >> > Split
> >> > > Reader API
> >> > >
> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> >> > > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
> >> > >
> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> >> > > [refactor][connectors] Backport of the connector-base exception
> >> handling
> >> > > from the Kafka Connector Pull Request
> >> > >
> >> > >
> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]>
> >> wrote:
> >> > >
> >> > > > I would love to see this FLIP-27 source interface improvement [1]
> >> made
> >> > to
> >> > > > 1.11.3.
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >> > > >
> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> >> > > [hidden email]>
> >> > > > wrote:
> >> > > >
> >> > > > > Thanks for the replies so far!
> >> > > > >
> >> > > > > Just to provide a brief update on the status of blockers for
> >> 1.11.3
> >> > so
> >> > > > far:
> >> > > > >
> >> > > > >
> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> >> > SourceReaderBase.pollNext
> >> > > > may
> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> >> > > > > https://github.com/apache/flink/pull/13776)
> >> > > > >
> >> > > > > *PR opened, reviewed + close to being merged*
> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw
> keyed
> >> > > stream
> >> > > > > if it isn't the writer (
> >> https://github.com/apache/flink/pull/13761)
> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> >> > skipping
> >> > > > > unwritten key groups (
> https://github.com/apache/flink/pull/13772)
> >> > > > >
> >> > > > > *Merged*
> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> >> suspended
> >> > > > > ZooKeeper connection
> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >> > > > >
> >> > > > > Right now as it seems, progress is mainly blocked on a reviewer
> >> for
> >> > > > > FLINK-19717.
> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1]
> to
> >> see
> >> > > if
> >> > > > > there are any fixes that should be applied to `release-1.11`.
> >> > > > >
> >> > > > > This is also a reminder, that if there are other blockers that
> we
> >> > need
> >> > > to
> >> > > > > be aware of, or a need to re-establish estimated time for
> getting
> >> > fixes
> >> > > > in
> >> > > > > and delay the RC for 1.11.3, please do let us know!
> >> > > > >
> >> > > > > Cheers,
> >> > > > > Gordon
> >> > > > >
> >> > > > > [1]
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >> > > > >
> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> >> [hidden email]>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> >> important
> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> >> > > > > >
> >> > > > > > Cheers,
> >> > > > > > Kostas
> >> > > > > >
> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]>
> wrote:
> >> > > > > > >
> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong
> >> for
> >> > > > > > > volunteering as our release managers.
> >> > > > > > >
> >> > > > > > > Best Regards,
> >> > > > > > > Yu
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> >> > [hidden email]>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Thanks Gordan for starting this discussion.
> >> > > > > > > > My pleasure to help with the release process.
> >> > > > > > > >
> >> > > > > > > > Thank you~
> >> > > > > > > >
> >> > > > > > > > Xintong Song
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> >> > > > [hidden email]
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks for starting this discussion Gordon. There are
> over
> >> > 100
> >> > > > > issues
> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
> 1.11.3
> >> > > > release.
> >> > > > > > Thanks
> >> > > > > > > > > for volunteering as our release managers Gordon and
> >> Xintong!
> >> > > > > > > > >
> >> > > > > > > > > Cheers,
> >> > > > > > > > > Till
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> >> > > > > > [hidden email]
> >> > > > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi,
> >> > > > > > > > >>
> >> > > > > > > > >> Xintong and I would like to start a discussion for
> >> releasing
> >> > > > Flink
> >> > > > > > > > 1.11.3
> >> > > > > > > > >> soon.
> >> > > > > > > > >>
> >> > > > > > > > >> It seems like we already have a few pressing issues
> that
> >> > needs
> >> > > > to
> >> > > > > be
> >> > > > > > > > >> included in a new hotfix release:
> >> > > > > > > > >>
> >> > > > > > > > >> - Heap-based timers’ restore behaviour is causing a
> >> > > critical
> >> > > > > > recovery
> >> > > > > > > > >> issue for StateFun [1] [2] [3].
> >> > > > > > > > >> - There are several robustness issues for the
> FLIP-27
> >> new
> >> > > > > source
> >> > > > > > API,
> >> > > > > > > > >> such as [4]. We already have some users using the
> >> FLIP-27
> >> > > API
> >> > > > > > with
> >> > > > > > > > >> 1.11.x,
> >> > > > > > > > >> so it would be important to get those fixes in for
> >> 1.11.x
> >> > > as
> >> > > > > > well.
> >> > > > > > > > >>
> >> > > > > > > > >> Apart from the issues that are already marked as
> blocker
> >> for
> >> > > > > 1.11.3
> >> > > > > > in
> >> > > > > > > > our
> >> > > > > > > > >> JIRA [5], please let us know in this thread if there is
> >> > > already
> >> > > > > > ongoing
> >> > > > > > > > >> work for other important fixes that we should try to
> >> > include.
> >> > > > > > > > >>
> >> > > > > > > > >> Xintong and I would like to volunteer for managing this
> >> > > release,
> >> > > > > and
> >> > > > > > > > will
> >> > > > > > > > >> try to communicate the priority of pending blockers
> over
> >> the
> >> > > > next
> >> > > > > > few
> >> > > > > > > > >> days.
> >> > > > > > > > >> Since the aforementioned issues are quite critical,
> we’d
> >> > like
> >> > > to
> >> > > > > aim
> >> > > > > > > > >> for a *feature
> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start
> the
> >> > > > release
> >> > > > > > voting
> >> > > > > > > > >> process the week after.
> >> > > > > > > > >> If that is too short of a notice and you might need
> more
> >> > time,
> >> > > > > > please
> >> > > > > > > > let
> >> > > > > > > > >> us know!
> >> > > > > > > > >>
> >> > > > > > > > >> Cheers,
> >> > > > > > > > >> Gordon
> >> > > > > > > > >>
> >> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> >> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> >> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> >> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> >> > > > > > > > >> [5]
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Best, Jingsong Lee
> >> >
> >>
> >
>

Steven Wu

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Basically, it would be great to get the latest code in the
flink-connector-files (FLIP-27).

On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]> wrote:

> Stephan, it will be great if we can also backport the DataStreamUtils
> related commits that help with collecting output from unbounded streams.
> e.g.
>
>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>
> I tried to copy and paste the code to unblock myself. but it quickly got
> into the rabbit hole of more and more code.
>
> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]> wrote:
>
>> I have started with backporting the source API changes. Some minor
>> conflicts to solve, will need a bit more to finish this.
>>
>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <[hidden email]>
>> wrote:
>>
>> > @Stephan Ewen <[hidden email]>
>> > Are there already plans or ongoing efforts for backporting the list of
>> > FLIP-27 changes that you posted?
>> >
>> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <[hidden email]>
>> > wrote:
>> >
>> >> Hi folks,
>> >>
>> >> Just to provide some updates concerning the status on the
>> >> test instabilities.
>> >>
>> >> Currently, we have 30 unresolved tickets labeled with `Affects Version`
>> >> 1.11.x.
>> >>
>> >>
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> >>
>> >> Among the 30 tickets, 11 of them are:
>> >> - Have occured in the recent 3 months
>> >> - Not confirmed to be pure testability issues
>> >> - Not confirmed to be rare condition cases
>> >>
>> >> It would be nice if someone familiar with these components can take a
>> look
>> >> into these issues.
>> >>
>> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
>> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
>> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
>> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
>> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
>> >> - https://issues.apache.org/jira/browse/FLINK-19775
>> >> (SystemProcessingTimeServiceTest)
>> >>
>> >> Thank you~
>> >>
>> >> Xintong Song
>> >>
>> >>
>> >>
>> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <[hidden email]>
>> >> wrote:
>> >>
>> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> >> >
>> >> > If possible, that would be great. Many people are looking forward to
>> the
>> >> > FLIP-27 interface, but they don't want to take the risk to upgrade to
>> >> 1.12
>> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
>> >> >
>> >> > Best,
>> >> > Jingsong
>> >> >
>> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]>
>> wrote:
>> >> >
>> >> > > Thanks for starting this.
>> >> > >
>> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>> >> > >
>> >> > > There were quite a few changes, and I think we need to cherry-pick
>> >> them
>> >> > all
>> >> > > to not get some inconsistent mix of changes and many merge
>> conflicts.
>> >> > > I made a list below of what we need to add to "release-1.11".
>> >> > >
>> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
>> >> > >
>> >> > > (Use: "git log
>> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> >> > >
>> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
>> and
>> >> > Split
>> >> > > Reader API
>> >> > >
>> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> >> > > [FLINK-19205][core] Add access to configuration and hostname in the
>> >> > > SourceReaderContext
>> >> > >
>> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
>> assign
>> >> > > single split
>> >> > >
>> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
>> >> > >
>> >> > > (Use: "git log flink-connectors/flink-connector-base")
>> >> > >
>> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
>> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> >> > >
>> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
>> >> condition
>> >> > > that would leave an inconsistent state
>> >> > >
>> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
>> and
>> >> > Split
>> >> > > Reader API
>> >> > >
>> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> >> > > [refactor][tests] Move some source test utils from
>> >> flink-connector-base
>> >> > to
>> >> > > flink-core
>> >> > >
>> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
>> >> > > "SplitReader.handleSplitsChanges()"
>> >> > >
>> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> >> > > [FLINK-19250][connectors] Fix error propagation in connector base
>> >> > > (SplitFetcherManager).
>> >> > >
>> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> >> > > [hotfix][connectors] Improve JavaDocs for
>> SingleThreadFetcherManager
>> >> > >
>> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> >> > > [FLINK-19225][connectors] Various small improvements to
>> >> SourceReaderBase
>> >> > > (part 2)
>> >> > >
>> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
>> >> > > elegantly.
>> >> > >
>> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> >> > > [hotfix][tests] Extend test coverage for
>> FutureCompletingBlockingQueue
>> >> > >
>> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> >> > > [FLINK-19245][connectors] Set default capacity for
>> >> > > FutureCompletingBlockingQueue.
>> >> > >
>> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in
>> Base
>> >> > > Connector
>> >> > >
>> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
>> >> availability
>> >> > > notifications.
>> >> > >
>> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> >> > > [refactor][core] Eagerly initialize the FetchTask to support proper
>> >> unit
>> >> > > testing
>> >> > >
>> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only
>> >> one
>> >> > > test into test method
>> >> > >
>> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> >> > > [FLINK-19225][connectors] Various small improvements to
>> >> SourceReaderBase
>> >> > >
>> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>> >> > >
>> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds
>> more
>> >> > > lightweight.
>> >> > >
>> >> > > commit e3d273de822b085183d09b275a445879ff94b350
>> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
>> RecordsWithSplitIds
>> >> to
>> >> > > support reuse of heavy objects.
>> >> > >
>> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for
>> the
>> >> > Split
>> >> > > Reader API
>> >> > >
>> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> >> > > [hotfix][connectors] Add RequestSplitEvent to
>> 'flink-connector-base'
>> >> > >
>> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> >> > > [refactor][connectors] Backport of the connector-base exception
>> >> handling
>> >> > > from the Kafka Connector Pull Request
>> >> > >
>> >> > >
>> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]>
>> >> wrote:
>> >> > >
>> >> > > > I would love to see this FLIP-27 source interface improvement [1]
>> >> made
>> >> > to
>> >> > > > 1.11.3.
>> >> > > >
>> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
>> >> > > >
>> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
>> >> > > [hidden email]>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Thanks for the replies so far!
>> >> > > > >
>> >> > > > > Just to provide a brief update on the status of blockers for
>> >> 1.11.3
>> >> > so
>> >> > > > far:
>> >> > > > >
>> >> > > > >
>> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> >> > SourceReaderBase.pollNext
>> >> > > > may
>> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
>> >> > > > > https://github.com/apache/flink/pull/13776)
>> >> > > > >
>> >> > > > > *PR opened, reviewed + close to being merged*
>> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw
>> keyed
>> >> > > stream
>> >> > > > > if it isn't the writer (
>> >> https://github.com/apache/flink/pull/13761)
>> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
>> >> > skipping
>> >> > > > > unwritten key groups (
>> https://github.com/apache/flink/pull/13772)
>> >> > > > >
>> >> > > > > *Merged*
>> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
>> >> suspended
>> >> > > > > ZooKeeper connection
>> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> >> > > > >
>> >> > > > > Right now as it seems, progress is mainly blocked on a reviewer
>> >> for
>> >> > > > > FLINK-19717.
>> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1]
>> to
>> >> see
>> >> > > if
>> >> > > > > there are any fixes that should be applied to `release-1.11`.
>> >> > > > >
>> >> > > > > This is also a reminder, that if there are other blockers that
>> we
>> >> > need
>> >> > > to
>> >> > > > > be aware of, or a need to re-establish estimated time for
>> getting
>> >> > fixes
>> >> > > > in
>> >> > > > > and delay the RC for 1.11.3, please do let us know!
>> >> > > > >
>> >> > > > > Cheers,
>> >> > > > > Gordon
>> >> > > > >
>> >> > > > > [1]
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> >> > > > >
>> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
>> >> [hidden email]>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
>> >> important
>> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
>> >> > > > > >
>> >> > > > > > Cheers,
>> >> > > > > > Kostas
>> >> > > > > >
>> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]>
>> wrote:
>> >> > > > > > >
>> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
>> Xintong
>> >> for
>> >> > > > > > > volunteering as our release managers.
>> >> > > > > > >
>> >> > > > > > > Best Regards,
>> >> > > > > > > Yu
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
>> >> > [hidden email]>
>> >> > > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > > Thanks Gordan for starting this discussion.
>> >> > > > > > > > My pleasure to help with the release process.
>> >> > > > > > > >
>> >> > > > > > > > Thank you~
>> >> > > > > > > >
>> >> > > > > > > > Xintong Song
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
>> >> > > > [hidden email]
>> >> > > > > >
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Thanks for starting this discussion Gordon. There are
>> over
>> >> > 100
>> >> > > > > issues
>> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
>> 1.11.3
>> >> > > > release.
>> >> > > > > > Thanks
>> >> > > > > > > > > for volunteering as our release managers Gordon and
>> >> Xintong!
>> >> > > > > > > > >
>> >> > > > > > > > > Cheers,
>> >> > > > > > > > > Till
>> >> > > > > > > > >
>> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
>> >> > > > > > [hidden email]
>> >> > > > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > >> Hi,
>> >> > > > > > > > >>
>> >> > > > > > > > >> Xintong and I would like to start a discussion for
>> >> releasing
>> >> > > > Flink
>> >> > > > > > > > 1.11.3
>> >> > > > > > > > >> soon.
>> >> > > > > > > > >>
>> >> > > > > > > > >> It seems like we already have a few pressing issues
>> that
>> >> > needs
>> >> > > > to
>> >> > > > > be
>> >> > > > > > > > >> included in a new hotfix release:
>> >> > > > > > > > >>
>> >> > > > > > > > >> - Heap-based timers’ restore behaviour is causing a
>> >> > > critical
>> >> > > > > > recovery
>> >> > > > > > > > >> issue for StateFun [1] [2] [3].
>> >> > > > > > > > >> - There are several robustness issues for the
>> FLIP-27
>> >> new
>> >> > > > > source
>> >> > > > > > API,
>> >> > > > > > > > >> such as [4]. We already have some users using the
>> >> FLIP-27
>> >> > > API
>> >> > > > > > with
>> >> > > > > > > > >> 1.11.x,
>> >> > > > > > > > >> so it would be important to get those fixes in for
>> >> 1.11.x
>> >> > > as
>> >> > > > > > well.
>> >> > > > > > > > >>
>> >> > > > > > > > >> Apart from the issues that are already marked as
>> blocker
>> >> for
>> >> > > > > 1.11.3
>> >> > > > > > in
>> >> > > > > > > > our
>> >> > > > > > > > >> JIRA [5], please let us know in this thread if there
>> is
>> >> > > already
>> >> > > > > > ongoing
>> >> > > > > > > > >> work for other important fixes that we should try to
>> >> > include.
>> >> > > > > > > > >>
>> >> > > > > > > > >> Xintong and I would like to volunteer for managing
>> this
>> >> > > release,
>> >> > > > > and
>> >> > > > > > > > will
>> >> > > > > > > > >> try to communicate the priority of pending blockers
>> over
>> >> the
>> >> > > > next
>> >> > > > > > few
>> >> > > > > > > > >> days.
>> >> > > > > > > > >> Since the aforementioned issues are quite critical,
>> we’d
>> >> > like
>> >> > > to
>> >> > > > > aim
>> >> > > > > > > > >> for a *feature
>> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start
>> the
>> >> > > > release
>> >> > > > > > voting
>> >> > > > > > > > >> process the week after.
>> >> > > > > > > > >> If that is too short of a notice and you might need
>> more
>> >> > time,
>> >> > > > > > please
>> >> > > > > > > > let
>> >> > > > > > > > >> us know!
>> >> > > > > > > > >>
>> >> > > > > > > > >> Cheers,
>> >> > > > > > > > >> Gordon
>> >> > > > > > > > >>
>> >> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
>> >> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
>> >> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
>> >> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
>> >> > > > > > > > >> [5]
>> >> > > > > > > > >>
>> >> > > > > > > > >>
>> >> > > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> >> > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > Best, Jingsong Lee
>> >> >
>> >>
>> >
>>
>

Stephan Ewen

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Hi Steven!

So far there are no plans to pick back the file system connector code. This
is still evolving and not finalized for 1.12, so I don't feel it is a good
candidate to be backported.
However, with the base connector changes backported, you should be able to
run the file connector code from master against 1.11.3.

The collect() utils can be picked back, I see no issue with that (it is
isolated utilities).

Best,
Stephan

On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <[hidden email]> wrote:

> Basically, it would be great to get the latest code in the
> flink-connector-files (FLIP-27).
>
> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]> wrote:
>
> > Stephan, it will be great if we can also backport the DataStreamUtils
> > related commits that help with collecting output from unbounded streams.
> > e.g.
> >
> >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >
> > I tried to copy and paste the code to unblock myself. but it quickly got
> > into the rabbit hole of more and more code.
> >
> > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]> wrote:
> >
> >> I have started with backporting the source API changes. Some minor
> >> conflicts to solve, will need a bit more to finish this.
> >>
> >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> [hidden email]>
> >> wrote:
> >>
> >> > @Stephan Ewen <[hidden email]>
> >> > Are there already plans or ongoing efforts for backporting the list of
> >> > FLIP-27 changes that you posted?
> >> >
> >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <[hidden email]>
> >> > wrote:
> >> >
> >> >> Hi folks,
> >> >>
> >> >> Just to provide some updates concerning the status on the
> >> >> test instabilities.
> >> >>
> >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> Version`
> >> >> 1.11.x.
> >> >>
> >> >>
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >> >>
> >> >> Among the 30 tickets, 11 of them are:
> >> >> - Have occured in the recent 3 months
> >> >> - Not confirmed to be pure testability issues
> >> >> - Not confirmed to be rare condition cases
> >> >>
> >> >> It would be nice if someone familiar with these components can take a
> >> look
> >> >> into these issues.
> >> >>
> >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> >> >> (SystemProcessingTimeServiceTest)
> >> >>
> >> >> Thank you~
> >> >>
> >> >> Xintong Song
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <[hidden email]
> >
> >> >> wrote:
> >> >>
> >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >> >> >
> >> >> > If possible, that would be great. Many people are looking forward
> to
> >> the
> >> >> > FLIP-27 interface, but they don't want to take the risk to upgrade
> to
> >> >> 1.12
> >> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> >> >> >
> >> >> > Best,
> >> >> > Jingsong
> >> >> >
> >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]>
> >> wrote:
> >> >> >
> >> >> > > Thanks for starting this.
> >> >> > >
> >> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> >> >> > >
> >> >> > > There were quite a few changes, and I think we need to
> cherry-pick
> >> >> them
> >> >> > all
> >> >> > > to not get some inconsistent mix of changes and many merge
> >> conflicts.
> >> >> > > I made a list below of what we need to add to "release-1.11".
> >> >> > >
> >> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> >> >> > >
> >> >> > > (Use: "git log
> >> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >> >> > >
> >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
> >> and
> >> >> > Split
> >> >> > > Reader API
> >> >> > >
> >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >> >> > > [FLINK-19205][core] Add access to configuration and hostname in
> the
> >> >> > > SourceReaderContext
> >> >> > >
> >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> >> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
> >> assign
> >> >> > > single split
> >> >> > >
> >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> >> >> > >
> >> >> > > (Use: "git log flink-connectors/flink-connector-base")
> >> >> > >
> >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >> >> > >
> >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
> >> >> condition
> >> >> > > that would leave an inconsistent state
> >> >> > >
> >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
> >> and
> >> >> > Split
> >> >> > > Reader API
> >> >> > >
> >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >> >> > > [refactor][tests] Move some source test utils from
> >> >> flink-connector-base
> >> >> > to
> >> >> > > flink-core
> >> >> > >
> >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> >> >> > > "SplitReader.handleSplitsChanges()"
> >> >> > >
> >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >> >> > > [FLINK-19250][connectors] Fix error propagation in connector base
> >> >> > > (SplitFetcherManager).
> >> >> > >
> >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> >> >> > > [hotfix][connectors] Improve JavaDocs for
> >> SingleThreadFetcherManager
> >> >> > >
> >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> >> >> > > [FLINK-19225][connectors] Various small improvements to
> >> >> SourceReaderBase
> >> >> > > (part 2)
> >> >> > >
> >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers
> more
> >> >> > > elegantly.
> >> >> > >
> >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >> >> > > [hotfix][tests] Extend test coverage for
> >> FutureCompletingBlockingQueue
> >> >> > >
> >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >> >> > > [FLINK-19245][connectors] Set default capacity for
> >> >> > > FutureCompletingBlockingQueue.
> >> >> > >
> >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in
> >> Base
> >> >> > > Connector
> >> >> > >
> >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> >> >> availability
> >> >> > > notifications.
> >> >> > >
> >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> >> >> > > [refactor][core] Eagerly initialize the FetchTask to support
> proper
> >> >> unit
> >> >> > > testing
> >> >> > >
> >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to
> only
> >> >> one
> >> >> > > test into test method
> >> >> > >
> >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> >> >> > > [FLINK-19225][connectors] Various small improvements to
> >> >> SourceReaderBase
> >> >> > >
> >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> elegantly.
> >> >> > >
> >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> >> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds
> >> more
> >> >> > > lightweight.
> >> >> > >
> >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> >> RecordsWithSplitIds
> >> >> to
> >> >> > > support reuse of heavy objects.
> >> >> > >
> >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for
> >> the
> >> >> > Split
> >> >> > > Reader API
> >> >> > >
> >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> >> 'flink-connector-base'
> >> >> > >
> >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> >> >> > > [refactor][connectors] Backport of the connector-base exception
> >> >> handling
> >> >> > > from the Kafka Connector Pull Request
> >> >> > >
> >> >> > >
> >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <[hidden email]>
> >> >> wrote:
> >> >> > >
> >> >> > > > I would love to see this FLIP-27 source interface improvement
> [1]
> >> >> made
> >> >> > to
> >> >> > > > 1.11.3.
> >> >> > > >
> >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >> >> > > >
> >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> >> >> > > [hidden email]>
> >> >> > > > wrote:
> >> >> > > >
> >> >> > > > > Thanks for the replies so far!
> >> >> > > > >
> >> >> > > > > Just to provide a brief update on the status of blockers for
> >> >> 1.11.3
> >> >> > so
> >> >> > > > far:
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> >> >> > SourceReaderBase.pollNext
> >> >> > > > may
> >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> >> >> > > > > https://github.com/apache/flink/pull/13776)
> >> >> > > > >
> >> >> > > > > *PR opened, reviewed + close to being merged*
> >> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw
> >> keyed
> >> >> > > stream
> >> >> > > > > if it isn't the writer (
> >> >> https://github.com/apache/flink/pull/13761)
> >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> >> >> > skipping
> >> >> > > > > unwritten key groups (
> >> https://github.com/apache/flink/pull/13772)
> >> >> > > > >
> >> >> > > > > *Merged*
> >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> >> >> suspended
> >> >> > > > > ZooKeeper connection
> >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >> >> > > > >
> >> >> > > > > Right now as it seems, progress is mainly blocked on a
> reviewer
> >> >> for
> >> >> > > > > FLINK-19717.
> >> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities
> [1]
> >> to
> >> >> see
> >> >> > > if
> >> >> > > > > there are any fixes that should be applied to `release-1.11`.
> >> >> > > > >
> >> >> > > > > This is also a reminder, that if there are other blockers
> that
> >> we
> >> >> > need
> >> >> > > to
> >> >> > > > > be aware of, or a need to re-establish estimated time for
> >> getting
> >> >> > fixes
> >> >> > > > in
> >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> >> >> > > > >
> >> >> > > > > Cheers,
> >> >> > > > > Gordon
> >> >> > > > >
> >> >> > > > > [1]
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >> >> > > > >
> >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> >> >> [hidden email]>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> >> >> important
> >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> >> >> > > > > >
> >> >> > > > > > Cheers,
> >> >> > > > > > Kostas
> >> >> > > > > >
> >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]>
> >> wrote:
> >> >> > > > > > >
> >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
> >> Xintong
> >> >> for
> >> >> > > > > > > volunteering as our release managers.
> >> >> > > > > > >
> >> >> > > > > > > Best Regards,
> >> >> > > > > > > Yu
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> >> >> > [hidden email]>
> >> >> > > > > > wrote:
> >> >> > > > > > >
> >> >> > > > > > > > Thanks Gordan for starting this discussion.
> >> >> > > > > > > > My pleasure to help with the release process.
> >> >> > > > > > > >
> >> >> > > > > > > > Thank you~
> >> >> > > > > > > >
> >> >> > > > > > > > Xintong Song
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> >> >> > > > [hidden email]
> >> >> > > > > >
> >> >> > > > > > > > wrote:
> >> >> > > > > > > >
> >> >> > > > > > > > > Thanks for starting this discussion Gordon. There are
> >> over
> >> >> > 100
> >> >> > > > > issues
> >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
> >> 1.11.3
> >> >> > > > release.
> >> >> > > > > > Thanks
> >> >> > > > > > > > > for volunteering as our release managers Gordon and
> >> >> Xintong!
> >> >> > > > > > > > >
> >> >> > > > > > > > > Cheers,
> >> >> > > > > > > > > Till
> >> >> > > > > > > > >
> >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> >> >> > > > > > [hidden email]
> >> >> > > > > > > > >
> >> >> > > > > > > > > wrote:
> >> >> > > > > > > > >
> >> >> > > > > > > > >> Hi,
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Xintong and I would like to start a discussion for
> >> >> releasing
> >> >> > > > Flink
> >> >> > > > > > > > 1.11.3
> >> >> > > > > > > > >> soon.
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> It seems like we already have a few pressing issues
> >> that
> >> >> > needs
> >> >> > > > to
> >> >> > > > > be
> >> >> > > > > > > > >> included in a new hotfix release:
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is
> causing a
> >> >> > > critical
> >> >> > > > > > recovery
> >> >> > > > > > > > >> issue for StateFun [1] [2] [3].
> >> >> > > > > > > > >> - There are several robustness issues for the
> >> FLIP-27
> >> >> new
> >> >> > > > > source
> >> >> > > > > > API,
> >> >> > > > > > > > >> such as [4]. We already have some users using the
> >> >> FLIP-27
> >> >> > > API
> >> >> > > > > > with
> >> >> > > > > > > > >> 1.11.x,
> >> >> > > > > > > > >> so it would be important to get those fixes in
> for
> >> >> 1.11.x
> >> >> > > as
> >> >> > > > > > well.
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Apart from the issues that are already marked as
> >> blocker
> >> >> for
> >> >> > > > > 1.11.3
> >> >> > > > > > in
> >> >> > > > > > > > our
> >> >> > > > > > > > >> JIRA [5], please let us know in this thread if there
> >> is
> >> >> > > already
> >> >> > > > > > ongoing
> >> >> > > > > > > > >> work for other important fixes that we should try to
> >> >> > include.
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Xintong and I would like to volunteer for managing
> >> this
> >> >> > > release,
> >> >> > > > > and
> >> >> > > > > > > > will
> >> >> > > > > > > > >> try to communicate the priority of pending blockers
> >> over
> >> >> the
> >> >> > > > next
> >> >> > > > > > few
> >> >> > > > > > > > >> days.
> >> >> > > > > > > > >> Since the aforementioned issues are quite critical,
> >> we’d
> >> >> > like
> >> >> > > to
> >> >> > > > > aim
> >> >> > > > > > > > >> for a *feature
> >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and
> start
> >> the
> >> >> > > > release
> >> >> > > > > > voting
> >> >> > > > > > > > >> process the week after.
> >> >> > > > > > > > >> If that is too short of a notice and you might need
> >> more
> >> >> > time,
> >> >> > > > > > please
> >> >> > > > > > > > let
> >> >> > > > > > > > >> us know!
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Cheers,
> >> >> > > > > > > > >> Gordon
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> [1]
> https://issues.apache.org/jira/browse/FLINK-19692
> >> >> > > > > > > > >> [2]
> https://issues.apache.org/jira/browse/FLINK-19741
> >> >> > > > > > > > >> [3]
> https://issues.apache.org/jira/browse/FLINK-19748
> >> >> > > > > > > > >> [4]
> https://issues.apache.org/jira/browse/FLINK-19717
> >> >> > > > > > > > >> [5]
> >> >> > > > > > > > >>
> >> >> > > > > > > > >>
> >> >> > > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >> >> > > > > > > > >>
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best, Jingsong Lee
> >> >> >
> >> >>
> >> >
> >>
> >
>

Steven Wu

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Stephan, thanks a lot for explaining the file connector. that makes sense.

I was asking because we were trying to reuse some of the implementations in
the file source for Iceberg source. Flink Iceberg source lives in the
Iceberg repo, which is not possible to code against the master branch of
the Flink code.

On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <[hidden email]> wrote:

> Hi Steven!
>
> So far there are no plans to pick back the file system connector code. This
> is still evolving and not finalized for 1.12, so I don't feel it is a good
> candidate to be backported.
> However, with the base connector changes backported, you should be able to
> run the file connector code from master against 1.11.3.
>
> The collect() utils can be picked back, I see no issue with that (it is
> isolated utilities).
>
> Best,
> Stephan
>
>
> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <[hidden email]> wrote:
>
> > Basically, it would be great to get the latest code in the
> > flink-connector-files (FLIP-27).
> >
> > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]> wrote:
> >
> > > Stephan, it will be great if we can also backport the DataStreamUtils
> > > related commits that help with collecting output from unbounded
> streams.
> > > e.g.
> > >
> > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >
> > > I tried to copy and paste the code to unblock myself. but it quickly
> got
> > > into the rabbit hole of more and more code.
> > >
> > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]>
> wrote:
> > >
> > >> I have started with backporting the source API changes. Some minor
> > >> conflicts to solve, will need a bit more to finish this.
> > >>
> > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > [hidden email]>
> > >> wrote:
> > >>
> > >> > @Stephan Ewen <[hidden email]>
> > >> > Are there already plans or ongoing efforts for backporting the list
> of
> > >> > FLIP-27 changes that you posted?
> > >> >
> > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <[hidden email]
> >
> > >> > wrote:
> > >> >
> > >> >> Hi folks,
> > >> >>
> > >> >> Just to provide some updates concerning the status on the
> > >> >> test instabilities.
> > >> >>
> > >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> > Version`
> > >> >> 1.11.x.
> > >> >>
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > >> >>
> > >> >> Among the 30 tickets, 11 of them are:
> > >> >> - Have occured in the recent 3 months
> > >> >> - Not confirmed to be pure testability issues
> > >> >> - Not confirmed to be rare condition cases
> > >> >>
> > >> >> It would be nice if someone familiar with these components can
> take a
> > >> look
> > >> >> into these issues.
> > >> >>
> > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> (BlobClientTest)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> (Format/Parquet)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > >> >> (SystemProcessingTimeServiceTest)
> > >> >>
> > >> >> Thank you~
> > >> >>
> > >> >> Xintong Song
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> [hidden email]
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > >> >> >
> > >> >> > If possible, that would be great. Many people are looking forward
> > to
> > >> the
> > >> >> > FLIP-27 interface, but they don't want to take the risk to
> upgrade
> > to
> > >> >> 1.12
> > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> > >> >> >
> > >> >> > Best,
> > >> >> > Jingsong
> > >> >> >
> > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]>
> > >> wrote:
> > >> >> >
> > >> >> > > Thanks for starting this.
> > >> >> > >
> > >> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> > >> >> > >
> > >> >> > > There were quite a few changes, and I think we need to
> > cherry-pick
> > >> >> them
> > >> >> > all
> > >> >> > > to not get some inconsistent mix of changes and many merge
> > >> conflicts.
> > >> >> > > I made a list below of what we need to add to "release-1.11".
> > >> >> > >
> > >> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> > >> >> > >
> > >> >> > > (Use: "git log
> > >> >> > >
> flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >> >> > >
> > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> API
> > >> and
> > >> >> > Split
> > >> >> > > Reader API
> > >> >> > >
> > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > >> >> > > [FLINK-19205][core] Add access to configuration and hostname in
> > the
> > >> >> > > SourceReaderContext
> > >> >> > >
> > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
> > >> assign
> > >> >> > > single split
> > >> >> > >
> > >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> > >> >> > >
> > >> >> > > (Use: "git log flink-connectors/flink-connector-base")
> > >> >> > >
> > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >> >> > >
> > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > >> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
> > >> >> condition
> > >> >> > > that would leave an inconsistent state
> > >> >> > >
> > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> API
> > >> and
> > >> >> > Split
> > >> >> > > Reader API
> > >> >> > >
> > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > >> >> > > [refactor][tests] Move some source test utils from
> > >> >> flink-connector-base
> > >> >> > to
> > >> >> > > flink-core
> > >> >> > >
> > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > >> >> > > "SplitReader.handleSplitsChanges()"
> > >> >> > >
> > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > >> >> > > [FLINK-19250][connectors] Fix error propagation in connector
> base
> > >> >> > > (SplitFetcherManager).
> > >> >> > >
> > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > >> SingleThreadFetcherManager
> > >> >> > >
> > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > >> >> SourceReaderBase
> > >> >> > > (part 2)
> > >> >> > >
> > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers
> > more
> > >> >> > > elegantly.
> > >> >> > >
> > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > >> >> > > [hotfix][tests] Extend test coverage for
> > >> FutureCompletingBlockingQueue
> > >> >> > >
> > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > >> >> > > FutureCompletingBlockingQueue.
> > >> >> > >
> > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in
> > >> Base
> > >> >> > > Connector
> > >> >> > >
> > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> > >> >> availability
> > >> >> > > notifications.
> > >> >> > >
> > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > >> >> > > [refactor][core] Eagerly initialize the FetchTask to support
> > proper
> > >> >> unit
> > >> >> > > testing
> > >> >> > >
> > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to
> > only
> > >> >> one
> > >> >> > > test into test method
> > >> >> > >
> > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > >> >> SourceReaderBase
> > >> >> > >
> > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > elegantly.
> > >> >> > >
> > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > >> >> > > [FLINK-18680][connectors] Make connector base
> RecordsWithSplitIds
> > >> more
> > >> >> > > lightweight.
> > >> >> > >
> > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > >> RecordsWithSplitIds
> > >> >> to
> > >> >> > > support reuse of heavy objects.
> > >> >> > >
> > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks
> for
> > >> the
> > >> >> > Split
> > >> >> > > Reader API
> > >> >> > >
> > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > >> 'flink-connector-base'
> > >> >> > >
> > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > >> >> > > [refactor][connectors] Backport of the connector-base exception
> > >> >> handling
> > >> >> > > from the Kafka Connector Pull Request
> > >> >> > >
> > >> >> > >
> > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> [hidden email]>
> > >> >> wrote:
> > >> >> > >
> > >> >> > > > I would love to see this FLIP-27 source interface improvement
> > [1]
> > >> >> made
> > >> >> > to
> > >> >> > > > 1.11.3.
> > >> >> > > >
> > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > >> >> > > >
> > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > >> >> > > [hidden email]>
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > > > Thanks for the replies so far!
> > >> >> > > > >
> > >> >> > > > > Just to provide a brief update on the status of blockers
> for
> > >> >> 1.11.3
> > >> >> > so
> > >> >> > > > far:
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > >> >> > SourceReaderBase.pollNext
> > >> >> > > > may
> > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > >> >> > > > >
> > >> >> > > > > *PR opened, reviewed + close to being merged*
> > >> >> > > > > - [FLINK-19741] Timer service should skip restoring from
> raw
> > >> keyed
> > >> >> > > stream
> > >> >> > > > > if it isn't the writer (
> > >> >> https://github.com/apache/flink/pull/13761)
> > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should
> be
> > >> >> > skipping
> > >> >> > > > > unwritten key groups (
> > >> https://github.com/apache/flink/pull/13772)
> > >> >> > > > >
> > >> >> > > > > *Merged*
> > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> > >> >> suspended
> > >> >> > > > > ZooKeeper connection
> > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >> >> > > > >
> > >> >> > > > > Right now as it seems, progress is mainly blocked on a
> > reviewer
> > >> >> for
> > >> >> > > > > FLINK-19717.
> > >> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities
> > [1]
> > >> to
> > >> >> see
> > >> >> > > if
> > >> >> > > > > there are any fixes that should be applied to
> `release-1.11`.
> > >> >> > > > >
> > >> >> > > > > This is also a reminder, that if there are other blockers
> > that
> > >> we
> > >> >> > need
> > >> >> > > to
> > >> >> > > > > be aware of, or a need to re-establish estimated time for
> > >> getting
> > >> >> > fixes
> > >> >> > > > in
> > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > >> >> > > > >
> > >> >> > > > > Cheers,
> > >> >> > > > > Gordon
> > >> >> > > > >
> > >> >> > > > > [1]
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >> >> > > > >
> > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > >> >> [hidden email]>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> > >> >> important
> > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > >> >> > > > > >
> > >> >> > > > > > Cheers,
> > >> >> > > > > > Kostas
> > >> >> > > > > >
> > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <[hidden email]>
> > >> wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
> > >> Xintong
> > >> >> for
> > >> >> > > > > > > volunteering as our release managers.
> > >> >> > > > > > >
> > >> >> > > > > > > Best Regards,
> > >> >> > > > > > > Yu
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > >> >> > [hidden email]>
> > >> >> > > > > > wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > >> >> > > > > > > > My pleasure to help with the release process.
> > >> >> > > > > > > >
> > >> >> > > > > > > > Thank you~
> > >> >> > > > > > > >
> > >> >> > > > > > > > Xintong Song
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > >> >> > > > [hidden email]
> > >> >> > > > > >
> > >> >> > > > > > > > wrote:
> > >> >> > > > > > > >
> > >> >> > > > > > > > > Thanks for starting this discussion Gordon. There
> are
> > >> over
> > >> >> > 100
> > >> >> > > > > issues
> > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
> > >> 1.11.3
> > >> >> > > > release.
> > >> >> > > > > > Thanks
> > >> >> > > > > > > > > for volunteering as our release managers Gordon and
> > >> >> Xintong!
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Cheers,
> > >> >> > > > > > > > > Till
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon)
> Tai <
> > >> >> > > > > > [hidden email]
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >> Hi,
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Xintong and I would like to start a discussion for
> > >> >> releasing
> > >> >> > > > Flink
> > >> >> > > > > > > > 1.11.3
> > >> >> > > > > > > > >> soon.
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> It seems like we already have a few pressing
> issues
> > >> that
> > >> >> > needs
> > >> >> > > > to
> > >> >> > > > > be
> > >> >> > > > > > > > >> included in a new hotfix release:
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is
> > causing a
> > >> >> > > critical
> > >> >> > > > > > recovery
> > >> >> > > > > > > > >> issue for StateFun [1] [2] [3].
> > >> >> > > > > > > > >> - There are several robustness issues for the
> > >> FLIP-27
> > >> >> new
> > >> >> > > > > source
> > >> >> > > > > > API,
> > >> >> > > > > > > > >> such as [4]. We already have some users using
> the
> > >> >> FLIP-27
> > >> >> > > API
> > >> >> > > > > > with
> > >> >> > > > > > > > >> 1.11.x,
> > >> >> > > > > > > > >> so it would be important to get those fixes in
> > for
> > >> >> 1.11.x
> > >> >> > > as
> > >> >> > > > > > well.
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Apart from the issues that are already marked as
> > >> blocker
> > >> >> for
> > >> >> > > > > 1.11.3
> > >> >> > > > > > in
> > >> >> > > > > > > > our
> > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if
> there
> > >> is
> > >> >> > > already
> > >> >> > > > > > ongoing
> > >> >> > > > > > > > >> work for other important fixes that we should try
> to
> > >> >> > include.
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Xintong and I would like to volunteer for managing
> > >> this
> > >> >> > > release,
> > >> >> > > > > and
> > >> >> > > > > > > > will
> > >> >> > > > > > > > >> try to communicate the priority of pending
> blockers
> > >> over
> > >> >> the
> > >> >> > > > next
> > >> >> > > > > > few
> > >> >> > > > > > > > >> days.
> > >> >> > > > > > > > >> Since the aforementioned issues are quite
> critical,
> > >> we’d
> > >> >> > like
> > >> >> > > to
> > >> >> > > > > aim
> > >> >> > > > > > > > >> for a *feature
> > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and
> > start
> > >> the
> > >> >> > > > release
> > >> >> > > > > > voting
> > >> >> > > > > > > > >> process the week after.
> > >> >> > > > > > > > >> If that is too short of a notice and you might
> need
> > >> more
> > >> >> > time,
> > >> >> > > > > > please
> > >> >> > > > > > > > let
> > >> >> > > > > > > > >> us know!
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Cheers,
> > >> >> > > > > > > > >> Gordon
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> [1]
> > https://issues.apache.org/jira/browse/FLINK-19692
> > >> >> > > > > > > > >> [2]
> > https://issues.apache.org/jira/browse/FLINK-19741
> > >> >> > > > > > > > >> [3]
> > https://issues.apache.org/jira/browse/FLINK-19748
> > >> >> > > > > > > > >> [4]
> > https://issues.apache.org/jira/browse/FLINK-19717
> > >> >> > > > > > > > >> [5]
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >>
> > >> >> > > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > Best, Jingsong Lee
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>

Stephan Ewen

Re: [DISCUSS] Releasing Apache Flink 1.11.3

@Steven would it be possible to initially copy some of the code into the
iceberg source and later replace it by a dependency on the Flink file
source?

On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <[hidden email]> wrote:

> Stephan, thanks a lot for explaining the file connector. that makes sense.
>
> I was asking because we were trying to reuse some of the implementations in
> the file source for Iceberg source. Flink Iceberg source lives in the
> Iceberg repo, which is not possible to code against the master branch of
> the Flink code.
>
> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <[hidden email]> wrote:
>
> > Hi Steven!
> >
> > So far there are no plans to pick back the file system connector code.
> This
> > is still evolving and not finalized for 1.12, so I don't feel it is a
> good
> > candidate to be backported.
> > However, with the base connector changes backported, you should be able
> to
> > run the file connector code from master against 1.11.3.
> >
> > The collect() utils can be picked back, I see no issue with that (it is
> > isolated utilities).
> >
> > Best,
> > Stephan
> >
> >
> > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <[hidden email]> wrote:
> >
> > > Basically, it would be great to get the latest code in the
> > > flink-connector-files (FLIP-27).
> > >
> > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]>
> wrote:
> > >
> > > > Stephan, it will be great if we can also backport the DataStreamUtils
> > > > related commits that help with collecting output from unbounded
> > streams.
> > > > e.g.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >
> > > > I tried to copy and paste the code to unblock myself. but it quickly
> > got
> > > > into the rabbit hole of more and more code.
> > > >
> > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]>
> > wrote:
> > > >
> > > >> I have started with backporting the source API changes. Some minor
> > > >> conflicts to solve, will need a bit more to finish this.
> > > >>
> > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > [hidden email]>
> > > >> wrote:
> > > >>
> > > >> > @Stephan Ewen <[hidden email]>
> > > >> > Are there already plans or ongoing efforts for backporting the
> list
> > of
> > > >> > FLIP-27 changes that you posted?
> > > >> >
> > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> [hidden email]
> > >
> > > >> > wrote:
> > > >> >
> > > >> >> Hi folks,
> > > >> >>
> > > >> >> Just to provide some updates concerning the status on the
> > > >> >> test instabilities.
> > > >> >>
> > > >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> > > Version`
> > > >> >> 1.11.x.
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > >> >>
> > > >> >> Among the 30 tickets, 11 of them are:
> > > >> >> - Have occured in the recent 3 months
> > > >> >> - Not confirmed to be pure testability issues
> > > >> >> - Not confirmed to be rare condition cases
> > > >> >>
> > > >> >> It would be nice if someone familiar with these components can
> > take a
> > > >> look
> > > >> >> into these issues.
> > > >> >>
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > (BlobClientTest)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > (Format/Parquet)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > >> >> (SystemProcessingTimeServiceTest)
> > > >> >>
> > > >> >> Thank you~
> > > >> >>
> > > >> >> Xintong Song
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > [hidden email]
> > > >
> > > >> >> wrote:
> > > >> >>
> > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > >> >> >
> > > >> >> > If possible, that would be great. Many people are looking
> forward
> > > to
> > > >> the
> > > >> >> > FLIP-27 interface, but they don't want to take the risk to
> > upgrade
> > > to
> > > >> >> 1.12
> > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> version.
> > > >> >> >
> > > >> >> > Best,
> > > >> >> > Jingsong
> > > >> >> >
> > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <[hidden email]
> >
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Thanks for starting this.
> > > >> >> > >
> > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> 1.11.x.
> > > >> >> > >
> > > >> >> > > There were quite a few changes, and I think we need to
> > > cherry-pick
> > > >> >> them
> > > >> >> > all
> > > >> >> > > to not get some inconsistent mix of changes and many merge
> > > >> conflicts.
> > > >> >> > > I made a list below of what we need to add to "release-1.11".
> > > >> >> > >
> > > >> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> > > >> >> > >
> > > >> >> > > (Use: "git log
> > > >> >> > >
> > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > >> >> > >
> > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> > API
> > > >> and
> > > >> >> > Split
> > > >> >> > > Reader API
> > > >> >> > >
> > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > >> >> > > [FLINK-19205][core] Add access to configuration and hostname
> in
> > > the
> > > >> >> > > SourceReaderContext
> > > >> >> > >
> > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods
> to
> > > >> assign
> > > >> >> > > single split
> > > >> >> > >
> > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> > > >> >> > >
> > > >> >> > > (Use: "git log flink-connectors/flink-connector-base")
> > > >> >> > >
> > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > >> >> > >
> > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> un-expected
> > > >> >> condition
> > > >> >> > > that would leave an inconsistent state
> > > >> >> > >
> > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> > API
> > > >> and
> > > >> >> > Split
> > > >> >> > > Reader API
> > > >> >> > >
> > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > >> >> > > [refactor][tests] Move some source test utils from
> > > >> >> flink-connector-base
> > > >> >> > to
> > > >> >> > > flink-core
> > > >> >> > >
> > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > >> >> > >
> > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > >> >> > > [FLINK-19250][connectors] Fix error propagation in connector
> > base
> > > >> >> > > (SplitFetcherManager).
> > > >> >> > >
> > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > >> SingleThreadFetcherManager
> > > >> >> > >
> > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > >> >> SourceReaderBase
> > > >> >> > > (part 2)
> > > >> >> > >
> > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> SplitFetchers
> > > more
> > > >> >> > > elegantly.
> > > >> >> > >
> > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > >> >> > > [hotfix][tests] Extend test coverage for
> > > >> FutureCompletingBlockingQueue
> > > >> >> > >
> > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > >> >> > > FutureCompletingBlockingQueue.
> > > >> >> > >
> > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model
> in
> > > >> Base
> > > >> >> > > Connector
> > > >> >> > >
> > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> > > >> >> availability
> > > >> >> > > notifications.
> > > >> >> > >
> > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to support
> > > proper
> > > >> >> unit
> > > >> >> > > testing
> > > >> >> > >
> > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant
> to
> > > only
> > > >> >> one
> > > >> >> > > test into test method
> > > >> >> > >
> > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > >> >> SourceReaderBase
> > > >> >> > >
> > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > > elegantly.
> > > >> >> > >
> > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > >> >> > > [FLINK-18680][connectors] Make connector base
> > RecordsWithSplitIds
> > > >> more
> > > >> >> > > lightweight.
> > > >> >> > >
> > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > >> RecordsWithSplitIds
> > > >> >> to
> > > >> >> > > support reuse of heavy objects.
> > > >> >> > >
> > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks
> > for
> > > >> the
> > > >> >> > Split
> > > >> >> > > Reader API
> > > >> >> > >
> > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > >> 'flink-connector-base'
> > > >> >> > >
> > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > >> >> > > [refactor][connectors] Backport of the connector-base
> exception
> > > >> >> handling
> > > >> >> > > from the Kafka Connector Pull Request
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > [hidden email]>
> > > >> >> wrote:
> > > >> >> > >
> > > >> >> > > > I would love to see this FLIP-27 source interface
> improvement
> > > [1]
> > > >> >> made
> > > >> >> > to
> > > >> >> > > > 1.11.3.
> > > >> >> > > >
> > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > >> >> > > >
> > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > >> >> > > [hidden email]>
> > > >> >> > > > wrote:
> > > >> >> > > >
> > > >> >> > > > > Thanks for the replies so far!
> > > >> >> > > > >
> > > >> >> > > > > Just to provide a brief update on the status of blockers
> > for
> > > >> >> 1.11.3
> > > >> >> > so
> > > >> >> > > > far:
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > >> >> > SourceReaderBase.pollNext
> > > >> >> > > > may
> > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > >> >> > > > >
> > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring from
> > raw
> > > >> keyed
> > > >> >> > > stream
> > > >> >> > > > > if it isn't the writer (
> > > >> >> https://github.com/apache/flink/pull/13761)
> > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
> should
> > be
> > > >> >> > skipping
> > > >> >> > > > > unwritten key groups (
> > > >> https://github.com/apache/flink/pull/13772)
> > > >> >> > > > >
> > > >> >> > > > > *Merged*
> > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case
> of
> > > >> >> suspended
> > > >> >> > > > > ZooKeeper connection
> > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >> >> > > > >
> > > >> >> > > > > Right now as it seems, progress is mainly blocked on a
> > > reviewer
> > > >> >> for
> > > >> >> > > > > FLINK-19717.
> > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> instabilities
> > > [1]
> > > >> to
> > > >> >> see
> > > >> >> > > if
> > > >> >> > > > > there are any fixes that should be applied to
> > `release-1.11`.
> > > >> >> > > > >
> > > >> >> > > > > This is also a reminder, that if there are other blockers
> > > that
> > > >> we
> > > >> >> > need
> > > >> >> > > to
> > > >> >> > > > > be aware of, or a need to re-establish estimated time for
> > > >> getting
> > > >> >> > fixes
> > > >> >> > > > in
> > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > > >> >> > > > >
> > > >> >> > > > > Cheers,
> > > >> >> > > > > Gordon
> > > >> >> > > > >
> > > >> >> > > > > [1]
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >> >> > > > >
> > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > > >> >> [hidden email]>
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number
> of
> > > >> >> important
> > > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > >> >> > > > > >
> > > >> >> > > > > > Cheers,
> > > >> >> > > > > > Kostas
> > > >> >> > > > > >
> > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> [hidden email]>
> > > >> wrote:
> > > >> >> > > > > > >
> > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
> > > >> Xintong
> > > >> >> for
> > > >> >> > > > > > > volunteering as our release managers.
> > > >> >> > > > > > >
> > > >> >> > > > > > > Best Regards,
> > > >> >> > > > > > > Yu
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > > >> >> > [hidden email]>
> > > >> >> > > > > > wrote:
> > > >> >> > > > > > >
> > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > > >> >> > > > > > > > My pleasure to help with the release process.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Thank you~
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Xintong Song
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > >> >> > > > [hidden email]
> > > >> >> > > > > >
> > > >> >> > > > > > > > wrote:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > > Thanks for starting this discussion Gordon. There
> > are
> > > >> over
> > > >> >> > 100
> > > >> >> > > > > issues
> > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> soonish
> > > >> 1.11.3
> > > >> >> > > > release.
> > > >> >> > > > > > Thanks
> > > >> >> > > > > > > > > for volunteering as our release managers Gordon
> and
> > > >> >> Xintong!
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Cheers,
> > > >> >> > > > > > > > > Till
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon)
> > Tai <
> > > >> >> > > > > > [hidden email]
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > wrote:
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >> Hi,
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Xintong and I would like to start a discussion
> for
> > > >> >> releasing
> > > >> >> > > > Flink
> > > >> >> > > > > > > > 1.11.3
> > > >> >> > > > > > > > >> soon.
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> It seems like we already have a few pressing
> > issues
> > > >> that
> > > >> >> > needs
> > > >> >> > > > to
> > > >> >> > > > > be
> > > >> >> > > > > > > > >> included in a new hotfix release:
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is
> > > causing a
> > > >> >> > > critical
> > > >> >> > > > > > recovery
> > > >> >> > > > > > > > >> issue for StateFun [1] [2] [3].
> > > >> >> > > > > > > > >> - There are several robustness issues for the
> > > >> FLIP-27
> > > >> >> new
> > > >> >> > > > > source
> > > >> >> > > > > > API,
> > > >> >> > > > > > > > >> such as [4]. We already have some users using
> > the
> > > >> >> FLIP-27
> > > >> >> > > API
> > > >> >> > > > > > with
> > > >> >> > > > > > > > >> 1.11.x,
> > > >> >> > > > > > > > >> so it would be important to get those fixes
> in
> > > for
> > > >> >> 1.11.x
> > > >> >> > > as
> > > >> >> > > > > > well.
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Apart from the issues that are already marked as
> > > >> blocker
> > > >> >> for
> > > >> >> > > > > 1.11.3
> > > >> >> > > > > > in
> > > >> >> > > > > > > > our
> > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if
> > there
> > > >> is
> > > >> >> > > already
> > > >> >> > > > > > ongoing
> > > >> >> > > > > > > > >> work for other important fixes that we should
> try
> > to
> > > >> >> > include.
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> managing
> > > >> this
> > > >> >> > > release,
> > > >> >> > > > > and
> > > >> >> > > > > > > > will
> > > >> >> > > > > > > > >> try to communicate the priority of pending
> > blockers
> > > >> over
> > > >> >> the
> > > >> >> > > > next
> > > >> >> > > > > > few
> > > >> >> > > > > > > > >> days.
> > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> > critical,
> > > >> we’d
> > > >> >> > like
> > > >> >> > > to
> > > >> >> > > > > aim
> > > >> >> > > > > > > > >> for a *feature
> > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and
> > > start
> > > >> the
> > > >> >> > > > release
> > > >> >> > > > > > voting
> > > >> >> > > > > > > > >> process the week after.
> > > >> >> > > > > > > > >> If that is too short of a notice and you might
> > need
> > > >> more
> > > >> >> > time,
> > > >> >> > > > > > please
> > > >> >> > > > > > > > let
> > > >> >> > > > > > > > >> us know!
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Cheers,
> > > >> >> > > > > > > > >> Gordon
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> [1]
> > > https://issues.apache.org/jira/browse/FLINK-19692
> > > >> >> > > > > > > > >> [2]
> > > https://issues.apache.org/jira/browse/FLINK-19741
> > > >> >> > > > > > > > >> [3]
> > > https://issues.apache.org/jira/browse/FLINK-19748
> > > >> >> > > > > > > > >> [4]
> > > https://issues.apache.org/jira/browse/FLINK-19717
> > > >> >> > > > > > > > >> [5]
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> > Best, Jingsong Lee
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>

Steven Wu

Re: [DISCUSS] Releasing Apache Flink 1.11.3

@Stephan Ewen <[hidden email]> yeah, we can do that. don't worry about
it. your earlier email had the perfect explanation on why file source
shouldn't be backported.

On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <[hidden email]> wrote:

> @Steven would it be possible to initially copy some of the code into the
> iceberg source and later replace it by a dependency on the Flink file
> source?
>
> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <[hidden email]> wrote:
>
> > Stephan, thanks a lot for explaining the file connector. that makes
> sense.
> >
> > I was asking because we were trying to reuse some of the implementations
> in
> > the file source for Iceberg source. Flink Iceberg source lives in the
> > Iceberg repo, which is not possible to code against the master branch of
> > the Flink code.
> >
> > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <[hidden email]> wrote:
> >
> > > Hi Steven!
> > >
> > > So far there are no plans to pick back the file system connector code.
> > This
> > > is still evolving and not finalized for 1.12, so I don't feel it is a
> > good
> > > candidate to be backported.
> > > However, with the base connector changes backported, you should be able
> > to
> > > run the file connector code from master against 1.11.3.
> > >
> > > The collect() utils can be picked back, I see no issue with that (it is
> > > isolated utilities).
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <[hidden email]> wrote:
> > >
> > > > Basically, it would be great to get the latest code in the
> > > > flink-connector-files (FLIP-27).
> > > >
> > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]>
> > wrote:
> > > >
> > > > > Stephan, it will be great if we can also backport the
> DataStreamUtils
> > > > > related commits that help with collecting output from unbounded
> > > streams.
> > > > > e.g.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > >
> > > > > I tried to copy and paste the code to unblock myself. but it
> quickly
> > > got
> > > > > into the rabbit hole of more and more code.
> > > > >
> > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]>
> > > wrote:
> > > > >
> > > > >> I have started with backporting the source API changes. Some minor
> > > > >> conflicts to solve, will need a bit more to finish this.
> > > > >>
> > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > > [hidden email]>
> > > > >> wrote:
> > > > >>
> > > > >> > @Stephan Ewen <[hidden email]>
> > > > >> > Are there already plans or ongoing efforts for backporting the
> > list
> > > of
> > > > >> > FLIP-27 changes that you posted?
> > > > >> >
> > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > [hidden email]
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Hi folks,
> > > > >> >>
> > > > >> >> Just to provide some updates concerning the status on the
> > > > >> >> test instabilities.
> > > > >> >>
> > > > >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> > > > Version`
> > > > >> >> 1.11.x.
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > >> >>
> > > > >> >> Among the 30 tickets, 11 of them are:
> > > > >> >> - Have occured in the recent 3 months
> > > > >> >> - Not confirmed to be pure testability issues
> > > > >> >> - Not confirmed to be rare condition cases
> > > > >> >>
> > > > >> >> It would be nice if someone familiar with these components can
> > > take a
> > > > >> look
> > > > >> >> into these issues.
> > > > >> >>
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > > (BlobClientTest)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > > (Format/Parquet)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > > >> >> (SystemProcessingTimeServiceTest)
> > > > >> >>
> > > > >> >> Thank you~
> > > > >> >>
> > > > >> >> Xintong Song
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > [hidden email]
> > > > >
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > > >> >> >
> > > > >> >> > If possible, that would be great. Many people are looking
> > forward
> > > > to
> > > > >> the
> > > > >> >> > FLIP-27 interface, but they don't want to take the risk to
> > > upgrade
> > > > to
> > > > >> >> 1.12
> > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> > version.
> > > > >> >> >
> > > > >> >> > Best,
> > > > >> >> > Jingsong
> > > > >> >> >
> > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> [hidden email]
> > >
> > > > >> wrote:
> > > > >> >> >
> > > > >> >> > > Thanks for starting this.
> > > > >> >> > >
> > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> > 1.11.x.
> > > > >> >> > >
> > > > >> >> > > There were quite a few changes, and I think we need to
> > > > cherry-pick
> > > > >> >> them
> > > > >> >> > all
> > > > >> >> > > to not get some inconsistent mix of changes and many merge
> > > > >> conflicts.
> > > > >> >> > > I made a list below of what we need to add to
> "release-1.11".
> > > > >> >> > >
> > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
> order)*
> > > > >> >> > >
> > > > >> >> > > (Use: "git log
> > > > >> >> > >
> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > >> >> > >
> > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> Source
> > > API
> > > > >> and
> > > > >> >> > Split
> > > > >> >> > > Reader API
> > > > >> >> > >
> > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > >> >> > > [FLINK-19205][core] Add access to configuration and
> hostname
> > in
> > > > the
> > > > >> >> > > SourceReaderContext
> > > > >> >> > >
> > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods
> > to
> > > > >> assign
> > > > >> >> > > single split
> > > > >> >> > >
> > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
> order)*
> > > > >> >> > >
> > > > >> >> > > (Use: "git log flink-connectors/flink-connector-base")
> > > > >> >> > >
> > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > >> >> > >
> > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> > un-expected
> > > > >> >> condition
> > > > >> >> > > that would leave an inconsistent state
> > > > >> >> > >
> > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> Source
> > > API
> > > > >> and
> > > > >> >> > Split
> > > > >> >> > > Reader API
> > > > >> >> > >
> > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > >> >> > > [refactor][tests] Move some source test utils from
> > > > >> >> flink-connector-base
> > > > >> >> > to
> > > > >> >> > > flink-core
> > > > >> >> > >
> > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > >> >> > >
> > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> connector
> > > base
> > > > >> >> > > (SplitFetcherManager).
> > > > >> >> > >
> > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > >> SingleThreadFetcherManager
> > > > >> >> > >
> > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > >> >> SourceReaderBase
> > > > >> >> > > (part 2)
> > > > >> >> > >
> > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > SplitFetchers
> > > > more
> > > > >> >> > > elegantly.
> > > > >> >> > >
> > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > >> FutureCompletingBlockingQueue
> > > > >> >> > >
> > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > > >> >> > > FutureCompletingBlockingQueue.
> > > > >> >> > >
> > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
> Model
> > in
> > > > >> Base
> > > > >> >> > > Connector
> > > > >> >> > >
> > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead
> to
> > > > >> >> availability
> > > > >> >> > > notifications.
> > > > >> >> > >
> > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
> support
> > > > proper
> > > > >> >> unit
> > > > >> >> > > testing
> > > > >> >> > >
> > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant
> > to
> > > > only
> > > > >> >> one
> > > > >> >> > > test into test method
> > > > >> >> > >
> > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > >> >> SourceReaderBase
> > > > >> >> > >
> > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > > > elegantly.
> > > > >> >> > >
> > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > RecordsWithSplitIds
> > > > >> more
> > > > >> >> > > lightweight.
> > > > >> >> > >
> > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > > >> RecordsWithSplitIds
> > > > >> >> to
> > > > >> >> > > support reuse of heavy objects.
> > > > >> >> > >
> > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
> mocks
> > > for
> > > > >> the
> > > > >> >> > Split
> > > > >> >> > > Reader API
> > > > >> >> > >
> > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > > >> 'flink-connector-base'
> > > > >> >> > >
> > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > > >> >> > > [refactor][connectors] Backport of the connector-base
> > exception
> > > > >> >> handling
> > > > >> >> > > from the Kafka Connector Pull Request
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > [hidden email]>
> > > > >> >> wrote:
> > > > >> >> > >
> > > > >> >> > > > I would love to see this FLIP-27 source interface
> > improvement
> > > > [1]
> > > > >> >> made
> > > > >> >> > to
> > > > >> >> > > > 1.11.3.
> > > > >> >> > > >
> > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > > >> >> > > >
> > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > > >> >> > > [hidden email]>
> > > > >> >> > > > wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Thanks for the replies so far!
> > > > >> >> > > > >
> > > > >> >> > > > > Just to provide a brief update on the status of
> blockers
> > > for
> > > > >> >> 1.11.3
> > > > >> >> > so
> > > > >> >> > > > far:
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > > >> >> > SourceReaderBase.pollNext
> > > > >> >> > > > may
> > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > > >> >> > > > >
> > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring
> from
> > > raw
> > > > >> keyed
> > > > >> >> > > stream
> > > > >> >> > > > > if it isn't the writer (
> > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
> > should
> > > be
> > > > >> >> > skipping
> > > > >> >> > > > > unwritten key groups (
> > > > >> https://github.com/apache/flink/pull/13772)
> > > > >> >> > > > >
> > > > >> >> > > > > *Merged*
> > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in
> case
> > of
> > > > >> >> suspended
> > > > >> >> > > > > ZooKeeper connection
> > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > >> >> > > > >
> > > > >> >> > > > > Right now as it seems, progress is mainly blocked on a
> > > > reviewer
> > > > >> >> for
> > > > >> >> > > > > FLINK-19717.
> > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > instabilities
> > > > [1]
> > > > >> to
> > > > >> >> see
> > > > >> >> > > if
> > > > >> >> > > > > there are any fixes that should be applied to
> > > `release-1.11`.
> > > > >> >> > > > >
> > > > >> >> > > > > This is also a reminder, that if there are other
> blockers
> > > > that
> > > > >> we
> > > > >> >> > need
> > > > >> >> > > to
> > > > >> >> > > > > be aware of, or a need to re-establish estimated time
> for
> > > > >> getting
> > > > >> >> > fixes
> > > > >> >> > > > in
> > > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > > > >> >> > > > >
> > > > >> >> > > > > Cheers,
> > > > >> >> > > > > Gordon
> > > > >> >> > > > >
> > > > >> >> > > > > [1]
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > >> >> > > > >
> > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > > > >> >> [hidden email]>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number
> > of
> > > > >> >> important
> > > > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Cheers,
> > > > >> >> > > > > > Kostas
> > > > >> >> > > > > >
> > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > [hidden email]>
> > > > >> wrote:
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon
> and
> > > > >> Xintong
> > > > >> >> for
> > > > >> >> > > > > > > volunteering as our release managers.
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Best Regards,
> > > > >> >> > > > > > > Yu
> > > > >> >> > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > > > >> >> > [hidden email]>
> > > > >> >> > > > > > wrote:
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > > > >> >> > > > > > > > My pleasure to help with the release process.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Thank you~
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Xintong Song
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > > >> >> > > > [hidden email]
> > > > >> >> > > > > >
> > > > >> >> > > > > > > > wrote:
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon.
> There
> > > are
> > > > >> over
> > > > >> >> > 100
> > > > >> >> > > > > issues
> > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> > soonish
> > > > >> 1.11.3
> > > > >> >> > > > release.
> > > > >> >> > > > > > Thanks
> > > > >> >> > > > > > > > > for volunteering as our release managers Gordon
> > and
> > > > >> >> Xintong!
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Cheers,
> > > > >> >> > > > > > > > > Till
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon)
> > > Tai <
> > > > >> >> > > > > > [hidden email]
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > wrote:
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > >> Hi,
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Xintong and I would like to start a discussion
> > for
> > > > >> >> releasing
> > > > >> >> > > > Flink
> > > > >> >> > > > > > > > 1.11.3
> > > > >> >> > > > > > > > >> soon.
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> It seems like we already have a few pressing
> > > issues
> > > > >> that
> > > > >> >> > needs
> > > > >> >> > > > to
> > > > >> >> > > > > be
> > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is
> > > > causing a
> > > > >> >> > > critical
> > > > >> >> > > > > > recovery
> > > > >> >> > > > > > > > >> issue for StateFun [1] [2] [3].
> > > > >> >> > > > > > > > >> - There are several robustness issues for
> the
> > > > >> FLIP-27
> > > > >> >> new
> > > > >> >> > > > > source
> > > > >> >> > > > > > API,
> > > > >> >> > > > > > > > >> such as [4]. We already have some users
> using
> > > the
> > > > >> >> FLIP-27
> > > > >> >> > > API
> > > > >> >> > > > > > with
> > > > >> >> > > > > > > > >> 1.11.x,
> > > > >> >> > > > > > > > >> so it would be important to get those fixes
> > in
> > > > for
> > > > >> >> 1.11.x
> > > > >> >> > > as
> > > > >> >> > > > > > well.
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Apart from the issues that are already marked
> as
> > > > >> blocker
> > > > >> >> for
> > > > >> >> > > > > 1.11.3
> > > > >> >> > > > > > in
> > > > >> >> > > > > > > > our
> > > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if
> > > there
> > > > >> is
> > > > >> >> > > already
> > > > >> >> > > > > > ongoing
> > > > >> >> > > > > > > > >> work for other important fixes that we should
> > try
> > > to
> > > > >> >> > include.
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> > managing
> > > > >> this
> > > > >> >> > > release,
> > > > >> >> > > > > and
> > > > >> >> > > > > > > > will
> > > > >> >> > > > > > > > >> try to communicate the priority of pending
> > > blockers
> > > > >> over
> > > > >> >> the
> > > > >> >> > > > next
> > > > >> >> > > > > > few
> > > > >> >> > > > > > > > >> days.
> > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> > > critical,
> > > > >> we’d
> > > > >> >> > like
> > > > >> >> > > to
> > > > >> >> > > > > aim
> > > > >> >> > > > > > > > >> for a *feature
> > > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)*
> and
> > > > start
> > > > >> the
> > > > >> >> > > > release
> > > > >> >> > > > > > voting
> > > > >> >> > > > > > > > >> process the week after.
> > > > >> >> > > > > > > > >> If that is too short of a notice and you might
> > > need
> > > > >> more
> > > > >> >> > time,
> > > > >> >> > > > > > please
> > > > >> >> > > > > > > > let
> > > > >> >> > > > > > > > >> us know!
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Cheers,
> > > > >> >> > > > > > > > >> Gordon
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> [1]
> > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > >> >> > > > > > > > >> [2]
> > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > >> >> > > > > > > > >> [3]
> > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > >> >> > > > > > > > >> [4]
> > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > >> >> > > > > > > > >> [5]
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > --
> > > > >> >> > Best, Jingsong Lee
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Tzu-Li (Gordon) Tai

Re: [DISCUSS] Releasing Apache Flink 1.11.3

> The collect() utils can be picked back, I see no issue with that (it is
isolated utilities).

Just checking on all the requested backports mentioned in this thread, and
figuring out which ones seem to still be unassigned / open.

Is someone working on backporting
https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
to release-1.11 at the moment?

On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <[hidden email]> wrote:

> @Stephan Ewen <[hidden email]> yeah, we can do that. don't worry about
> it. your earlier email had the perfect explanation on why file source
> shouldn't be backported.
>
> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <[hidden email]> wrote:
>
> > @Steven would it be possible to initially copy some of the code into the
> > iceberg source and later replace it by a dependency on the Flink file
> > source?
> >
> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <[hidden email]> wrote:
> >
> > > Stephan, thanks a lot for explaining the file connector. that makes
> > sense.
> > >
> > > I was asking because we were trying to reuse some of the
> implementations
> > in
> > > the file source for Iceberg source. Flink Iceberg source lives in the
> > > Iceberg repo, which is not possible to code against the master branch
> of
> > > the Flink code.
> > >
> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <[hidden email]> wrote:
> > >
> > > > Hi Steven!
> > > >
> > > > So far there are no plans to pick back the file system connector
> code.
> > > This
> > > > is still evolving and not finalized for 1.12, so I don't feel it is a
> > > good
> > > > candidate to be backported.
> > > > However, with the base connector changes backported, you should be
> able
> > > to
> > > > run the file connector code from master against 1.11.3.
> > > >
> > > > The collect() utils can be picked back, I see no issue with that (it
> is
> > > > isolated utilities).
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <[hidden email]>
> wrote:
> > > >
> > > > > Basically, it would be great to get the latest code in the
> > > > > flink-connector-files (FLIP-27).
> > > > >
> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Stephan, it will be great if we can also backport the
> > DataStreamUtils
> > > > > > related commits that help with collecting output from unbounded
> > > > streams.
> > > > > > e.g.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > >
> > > > > > I tried to copy and paste the code to unblock myself. but it
> > quickly
> > > > got
> > > > > > into the rabbit hole of more and more code.
> > > > > >
> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > >> I have started with backporting the source API changes. Some
> minor
> > > > > >> conflicts to solve, will need a bit more to finish this.
> > > > > >>
> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > > > [hidden email]>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > @Stephan Ewen <[hidden email]>
> > > > > >> > Are there already plans or ongoing efforts for backporting the
> > > list
> > > > of
> > > > > >> > FLIP-27 changes that you posted?
> > > > > >> >
> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > [hidden email]
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Hi folks,
> > > > > >> >>
> > > > > >> >> Just to provide some updates concerning the status on the
> > > > > >> >> test instabilities.
> > > > > >> >>
> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
> `Affects
> > > > > Version`
> > > > > >> >> 1.11.x.
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > > >> >>
> > > > > >> >> Among the 30 tickets, 11 of them are:
> > > > > >> >> - Have occured in the recent 3 months
> > > > > >> >> - Not confirmed to be pure testability issues
> > > > > >> >> - Not confirmed to be rare condition cases
> > > > > >> >>
> > > > > >> >> It would be nice if someone familiar with these components
> can
> > > > take a
> > > > > >> look
> > > > > >> >> into these issues.
> > > > > >> >>
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > > > (BlobClientTest)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > > > (Format/Parquet)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > > > >> >> (SystemProcessingTimeServiceTest)
> > > > > >> >>
> > > > > >> >> Thank you~
> > > > > >> >>
> > > > > >> >> Xintong Song
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > > [hidden email]
> > > > > >
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > > > >> >> >
> > > > > >> >> > If possible, that would be great. Many people are looking
> > > forward
> > > > > to
> > > > > >> the
> > > > > >> >> > FLIP-27 interface, but they don't want to take the risk to
> > > > upgrade
> > > > > to
> > > > > >> >> 1.12
> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> > > version.
> > > > > >> >> >
> > > > > >> >> > Best,
> > > > > >> >> > Jingsong
> > > > > >> >> >
> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > [hidden email]
> > > >
> > > > > >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Thanks for starting this.
> > > > > >> >> > >
> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> > > 1.11.x.
> > > > > >> >> > >
> > > > > >> >> > > There were quite a few changes, and I think we need to
> > > > > cherry-pick
> > > > > >> >> them
> > > > > >> >> > all
> > > > > >> >> > > to not get some inconsistent mix of changes and many
> merge
> > > > > >> conflicts.
> > > > > >> >> > > I made a list below of what we need to add to
> > "release-1.11".
> > > > > >> >> > >
> > > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
> > order)*
> > > > > >> >> > >
> > > > > >> >> > > (Use: "git log
> > > > > >> >> > >
> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > > >> >> > >
> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> > Source
> > > > API
> > > > > >> and
> > > > > >> >> > Split
> > > > > >> >> > > Reader API
> > > > > >> >> > >
> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
> > hostname
> > > in
> > > > > the
> > > > > >> >> > > SourceReaderContext
> > > > > >> >> > >
> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
> methods
> > > to
> > > > > >> assign
> > > > > >> >> > > single split
> > > > > >> >> > >
> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
> > order)*
> > > > > >> >> > >
> > > > > >> >> > > (Use: "git log flink-connectors/flink-connector-base")
> > > > > >> >> > >
> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > > >> >> > >
> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> > > un-expected
> > > > > >> >> condition
> > > > > >> >> > > that would leave an inconsistent state
> > > > > >> >> > >
> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> > Source
> > > > API
> > > > > >> and
> > > > > >> >> > Split
> > > > > >> >> > > Reader API
> > > > > >> >> > >
> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > > >> >> > > [refactor][tests] Move some source test utils from
> > > > > >> >> flink-connector-base
> > > > > >> >> > to
> > > > > >> >> > > flink-core
> > > > > >> >> > >
> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling
> in
> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > > >> >> > >
> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> > connector
> > > > base
> > > > > >> >> > > (SplitFetcherManager).
> > > > > >> >> > >
> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > > >> SingleThreadFetcherManager
> > > > > >> >> > >
> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > > >> >> SourceReaderBase
> > > > > >> >> > > (part 2)
> > > > > >> >> > >
> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > > SplitFetchers
> > > > > more
> > > > > >> >> > > elegantly.
> > > > > >> >> > >
> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > > >> FutureCompletingBlockingQueue
> > > > > >> >> > >
> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > > > >> >> > > FutureCompletingBlockingQueue.
> > > > > >> >> > >
> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
> > Model
> > > in
> > > > > >> Base
> > > > > >> >> > > Connector
> > > > > >> >> > >
> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead
> > to
> > > > > >> >> availability
> > > > > >> >> > > notifications.
> > > > > >> >> > >
> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
> > support
> > > > > proper
> > > > > >> >> unit
> > > > > >> >> > > testing
> > > > > >> >> > >
> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
> relevant
> > > to
> > > > > only
> > > > > >> >> one
> > > > > >> >> > > test into test method
> > > > > >> >> > >
> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > > >> >> SourceReaderBase
> > > > > >> >> > >
> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > > > > elegantly.
> > > > > >> >> > >
> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > > RecordsWithSplitIds
> > > > > >> more
> > > > > >> >> > > lightweight.
> > > > > >> >> > >
> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > > > >> RecordsWithSplitIds
> > > > > >> >> to
> > > > > >> >> > > support reuse of heavy objects.
> > > > > >> >> > >
> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
> > mocks
> > > > for
> > > > > >> the
> > > > > >> >> > Split
> > > > > >> >> > > Reader API
> > > > > >> >> > >
> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > > > >> 'flink-connector-base'
> > > > > >> >> > >
> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > > > >> >> > > [refactor][connectors] Backport of the connector-base
> > > exception
> > > > > >> >> handling
> > > > > >> >> > > from the Kafka Connector Pull Request
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > > [hidden email]>
> > > > > >> >> wrote:
> > > > > >> >> > >
> > > > > >> >> > > > I would love to see this FLIP-27 source interface
> > > improvement
> > > > > [1]
> > > > > >> >> made
> > > > > >> >> > to
> > > > > >> >> > > > 1.11.3.
> > > > > >> >> > > >
> > > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > > > >> >> > > >
> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > > > >> >> > > [hidden email]>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > > > Thanks for the replies so far!
> > > > > >> >> > > > >
> > > > > >> >> > > > > Just to provide a brief update on the status of
> > blockers
> > > > for
> > > > > >> >> 1.11.3
> > > > > >> >> > so
> > > > > >> >> > > > far:
> > > > > >> >> > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > > > >> >> > SourceReaderBase.pollNext
> > > > > >> >> > > > may
> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > > > >> >> > > > >
> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring
> > from
> > > > raw
> > > > > >> keyed
> > > > > >> >> > > stream
> > > > > >> >> > > > > if it isn't the writer (
> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
> > > should
> > > > be
> > > > > >> >> > skipping
> > > > > >> >> > > > > unwritten key groups (
> > > > > >> https://github.com/apache/flink/pull/13772)
> > > > > >> >> > > > >
> > > > > >> >> > > > > *Merged*
> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in
> > case
> > > of
> > > > > >> >> suspended
> > > > > >> >> > > > > ZooKeeper connection
> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > > >> >> > > > >
> > > > > >> >> > > > > Right now as it seems, progress is mainly blocked on
> a
> > > > > reviewer
> > > > > >> >> for
> > > > > >> >> > > > > FLINK-19717.
> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > > instabilities
> > > > > [1]
> > > > > >> to
> > > > > >> >> see
> > > > > >> >> > > if
> > > > > >> >> > > > > there are any fixes that should be applied to
> > > > `release-1.11`.
> > > > > >> >> > > > >
> > > > > >> >> > > > > This is also a reminder, that if there are other
> > blockers
> > > > > that
> > > > > >> we
> > > > > >> >> > need
> > > > > >> >> > > to
> > > > > >> >> > > > > be aware of, or a need to re-establish estimated time
> > for
> > > > > >> getting
> > > > > >> >> > fixes
> > > > > >> >> > > > in
> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > > > > >> >> > > > >
> > > > > >> >> > > > > Cheers,
> > > > > >> >> > > > > Gordon
> > > > > >> >> > > > >
> > > > > >> >> > > > > [1]
> > > > > >> >> > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > > >> >> > > > >
> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > > > > >> >> [hidden email]>
> > > > > >> >> > > > > wrote:
> > > > > >> >> > > > >
> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a
> number
> > > of
> > > > > >> >> important
> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> volunteering.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Cheers,
> > > > > >> >> > > > > > Kostas
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > > [hidden email]>
> > > > > >> wrote:
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon
> > and
> > > > > >> Xintong
> > > > > >> >> for
> > > > > >> >> > > > > > > volunteering as our release managers.
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > Best Regards,
> > > > > >> >> > > > > > > Yu
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > > > > >> >> > [hidden email]>
> > > > > >> >> > > > > > wrote:
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > > > > >> >> > > > > > > > My pleasure to help with the release process.
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Thank you~
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Xintong Song
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann
> <
> > > > > >> >> > > > [hidden email]
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > > > wrote:
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon.
> > There
> > > > are
> > > > > >> over
> > > > > >> >> > 100
> > > > > >> >> > > > > issues
> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> > > soonish
> > > > > >> 1.11.3
> > > > > >> >> > > > release.
> > > > > >> >> > > > > > Thanks
> > > > > >> >> > > > > > > > > for volunteering as our release managers
> Gordon
> > > and
> > > > > >> >> Xintong!
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > Cheers,
> > > > > >> >> > > > > > > > > Till
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
> (Gordon)
> > > > Tai <
> > > > > >> >> > > > > > [hidden email]
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > wrote:
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > >> Hi,
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
> discussion
> > > for
> > > > > >> >> releasing
> > > > > >> >> > > > Flink
> > > > > >> >> > > > > > > > 1.11.3
> > > > > >> >> > > > > > > > >> soon.
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> It seems like we already have a few pressing
> > > > issues
> > > > > >> that
> > > > > >> >> > needs
> > > > > >> >> > > > to
> > > > > >> >> > > > > be
> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> - Heap-based timers’ restore behaviour is
> > > > > causing a
> > > > > >> >> > > critical
> > > > > >> >> > > > > > recovery
> > > > > >> >> > > > > > > > >> issue for StateFun [1] [2] [3].
> > > > > >> >> > > > > > > > >> - There are several robustness issues for
> > the
> > > > > >> FLIP-27
> > > > > >> >> new
> > > > > >> >> > > > > source
> > > > > >> >> > > > > > API,
> > > > > >> >> > > > > > > > >> such as [4]. We already have some users
> > using
> > > > the
> > > > > >> >> FLIP-27
> > > > > >> >> > > API
> > > > > >> >> > > > > > with
> > > > > >> >> > > > > > > > >> 1.11.x,
> > > > > >> >> > > > > > > > >> so it would be important to get those
> fixes
> > > in
> > > > > for
> > > > > >> >> 1.11.x
> > > > > >> >> > > as
> > > > > >> >> > > > > > well.
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Apart from the issues that are already
> marked
> > as
> > > > > >> blocker
> > > > > >> >> for
> > > > > >> >> > > > > 1.11.3
> > > > > >> >> > > > > > in
> > > > > >> >> > > > > > > > our
> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread
> if
> > > > there
> > > > > >> is
> > > > > >> >> > > already
> > > > > >> >> > > > > > ongoing
> > > > > >> >> > > > > > > > >> work for other important fixes that we
> should
> > > try
> > > > to
> > > > > >> >> > include.
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> > > managing
> > > > > >> this
> > > > > >> >> > > release,
> > > > > >> >> > > > > and
> > > > > >> >> > > > > > > > will
> > > > > >> >> > > > > > > > >> try to communicate the priority of pending
> > > > blockers
> > > > > >> over
> > > > > >> >> the
> > > > > >> >> > > > next
> > > > > >> >> > > > > > few
> > > > > >> >> > > > > > > > >> days.
> > > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> > > > critical,
> > > > > >> we’d
> > > > > >> >> > like
> > > > > >> >> > > to
> > > > > >> >> > > > > aim
> > > > > >> >> > > > > > > > >> for a *feature
> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)*
> > and
> > > > > start
> > > > > >> the
> > > > > >> >> > > > release
> > > > > >> >> > > > > > voting
> > > > > >> >> > > > > > > > >> process the week after.
> > > > > >> >> > > > > > > > >> If that is too short of a notice and you
> might
> > > > need
> > > > > >> more
> > > > > >> >> > time,
> > > > > >> >> > > > > > please
> > > > > >> >> > > > > > > > let
> > > > > >> >> > > > > > > > >> us know!
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Cheers,
> > > > > >> >> > > > > > > > >> Gordon
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> [1]
> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > > >> >> > > > > > > > >> [2]
> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > > >> >> > > > > > > > >> [3]
> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > > >> >> > > > > > > > >> [4]
> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > > >> >> > > > > > > > >> [5]
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > --
> > > > > >> >> > Best, Jingsong Lee
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Tzu-Li (Gordon) Tai

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Thanks for the efforts so far with pushing for 1.11.3.

I'd like to provide a mid-week update on how we're looking with all the
pending blockers and backports:

*Blockers**:*

Currently all blockers either have an open PR, or have been merged. I'll
highlight below the blockers *that still require reviewing efforts for them
to move forward:*

- [FLINK-19909] Flink application in attach mode could not terminate when
the only job is canceled. PR: https://github.com/apache/flink/pull/13911
- [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
SplitReader.fetch throws. PR: https://github.com/apache/flink/pull/13776

The above PRs currently have no reviews at all yet. They seem to already
have designated reviewers.

*Backports:*
- FLIP-27 robustness improvement backports: Stephan is currently working on
backporting several FLIP-27 changes. There are no PRs yet for the
backported changes.
- DataStreamUtils.collect() refactorings backport: Steven Wu mentioned to
backport this, but AFAIK this isn't assigned to anyone yet.

*ETAs*:

I'd like to request ETAs for the remaining backports, to prevent a creep in
the scope of this bugfix release.

We already have Flink users that would benefit from fixes that have been
merged to release-1.11 already, so technically speaking the backports
should be considered "nice-to-have" (to the best of my knowledge of the
changes) and could potentially be moved to a follow-up 1.11.4.
Most notably, the Stateful Functions project is already waiting on Flink
1.11.3 to address critical recovery issues (please see the StateFun 2.2.1
release discussion thread [1]).

@Stephan Ewen <[hidden email]> @Becket Qin <[hidden email]> could
you provide an ETA for the FLIP-27 backports? It would help to get a better
estimate to decide how we proceed here.

Cheers,
Gordon

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html

On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> > The collect() utils can be picked back, I see no issue with that (it is
> isolated utilities).
>
> Just checking on all the requested backports mentioned in this thread, and
> figuring out which ones seem to still be unassigned / open.
>
> Is someone working on backporting
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> to release-1.11 at the moment?
>
> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <[hidden email]> wrote:
>
>> @Stephan Ewen <[hidden email]> yeah, we can do that. don't worry about
>> it. your earlier email had the perfect explanation on why file source
>> shouldn't be backported.
>>
>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <[hidden email]> wrote:
>>
>> > @Steven would it be possible to initially copy some of the code into the
>> > iceberg source and later replace it by a dependency on the Flink file
>> > source?
>> >
>> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <[hidden email]> wrote:
>> >
>> > > Stephan, thanks a lot for explaining the file connector. that makes
>> > sense.
>> > >
>> > > I was asking because we were trying to reuse some of the
>> implementations
>> > in
>> > > the file source for Iceberg source. Flink Iceberg source lives in the
>> > > Iceberg repo, which is not possible to code against the master branch
>> of
>> > > the Flink code.
>> > >
>> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <[hidden email]> wrote:
>> > >
>> > > > Hi Steven!
>> > > >
>> > > > So far there are no plans to pick back the file system connector
>> code.
>> > > This
>> > > > is still evolving and not finalized for 1.12, so I don't feel it is
>> a
>> > > good
>> > > > candidate to be backported.
>> > > > However, with the base connector changes backported, you should be
>> able
>> > > to
>> > > > run the file connector code from master against 1.11.3.
>> > > >
>> > > > The collect() utils can be picked back, I see no issue with that
>> (it is
>> > > > isolated utilities).
>> > > >
>> > > > Best,
>> > > > Stephan
>> > > >
>> > > >
>> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <[hidden email]>
>> wrote:
>> > > >
>> > > > > Basically, it would be great to get the latest code in the
>> > > > > flink-connector-files (FLIP-27).
>> > > > >
>> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <[hidden email]>
>> > > wrote:
>> > > > >
>> > > > > > Stephan, it will be great if we can also backport the
>> > DataStreamUtils
>> > > > > > related commits that help with collecting output from unbounded
>> > > > streams.
>> > > > > > e.g.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > > > >
>> > > > > > I tried to copy and paste the code to unblock myself. but it
>> > quickly
>> > > > got
>> > > > > > into the rabbit hole of more and more code.
>> > > > > >
>> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <[hidden email]
>> >
>> > > > wrote:
>> > > > > >
>> > > > > >> I have started with backporting the source API changes. Some
>> minor
>> > > > > >> conflicts to solve, will need a bit more to finish this.
>> > > > > >>
>> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
>> > > > > [hidden email]>
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >> > @Stephan Ewen <[hidden email]>
>> > > > > >> > Are there already plans or ongoing efforts for backporting
>> the
>> > > list
>> > > > of
>> > > > > >> > FLIP-27 changes that you posted?
>> > > > > >> >
>> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
>> > > [hidden email]
>> > > > >
>> > > > > >> > wrote:
>> > > > > >> >
>> > > > > >> >> Hi folks,
>> > > > > >> >>
>> > > > > >> >> Just to provide some updates concerning the status on the
>> > > > > >> >> test instabilities.
>> > > > > >> >>
>> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
>> `Affects
>> > > > > Version`
>> > > > > >> >> 1.11.x.
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> > > > > >> >>
>> > > > > >> >> Among the 30 tickets, 11 of them are:
>> > > > > >> >> - Have occured in the recent 3 months
>> > > > > >> >> - Not confirmed to be pure testability issues
>> > > > > >> >> - Not confirmed to be rare condition cases
>> > > > > >> >>
>> > > > > >> >> It would be nice if someone familiar with these components
>> can
>> > > > take a
>> > > > > >> look
>> > > > > >> >> into these issues.
>> > > > > >> >>
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
>> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
>> > > > (BlobClientTest)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
>> > > > (Format/Parquet)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
>> > > > > >> >> (SystemProcessingTimeServiceTest)
>> > > > > >> >>
>> > > > > >> >> Thank you~
>> > > > > >> >>
>> > > > > >> >> Xintong Song
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
>> > > > [hidden email]
>> > > > > >
>> > > > > >> >> wrote:
>> > > > > >> >>
>> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> > > > > >> >> >
>> > > > > >> >> > If possible, that would be great. Many people are looking
>> > > forward
>> > > > > to
>> > > > > >> the
>> > > > > >> >> > FLIP-27 interface, but they don't want to take the risk to
>> > > > upgrade
>> > > > > to
>> > > > > >> >> 1.12
>> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
>> > > version.
>> > > > > >> >> >
>> > > > > >> >> > Best,
>> > > > > >> >> > Jingsong
>> > > > > >> >> >
>> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
>> > [hidden email]
>> > > >
>> > > > > >> wrote:
>> > > > > >> >> >
>> > > > > >> >> > > Thanks for starting this.
>> > > > > >> >> > >
>> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
>> > > 1.11.x.
>> > > > > >> >> > >
>> > > > > >> >> > > There were quite a few changes, and I think we need to
>> > > > > cherry-pick
>> > > > > >> >> them
>> > > > > >> >> > all
>> > > > > >> >> > > to not get some inconsistent mix of changes and many
>> merge
>> > > > > >> conflicts.
>> > > > > >> >> > > I made a list below of what we need to add to
>> > "release-1.11".
>> > > > > >> >> > >
>> > > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
>> > order)*
>> > > > > >> >> > >
>> > > > > >> >> > > (Use: "git log
>> > > > > >> >> > >
>> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > > > > >> >> > >
>> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
>> > Source
>> > > > API
>> > > > > >> and
>> > > > > >> >> > Split
>> > > > > >> >> > > Reader API
>> > > > > >> >> > >
>> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
>> > hostname
>> > > in
>> > > > > the
>> > > > > >> >> > > SourceReaderContext
>> > > > > >> >> > >
>> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
>> methods
>> > > to
>> > > > > >> assign
>> > > > > >> >> > > single split
>> > > > > >> >> > >
>> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
>> > order)*
>> > > > > >> >> > >
>> > > > > >> >> > > (Use: "git log flink-connectors/flink-connector-base")
>> > > > > >> >> > >
>> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions
>> for
>> > > > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > > > > >> >> > >
>> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
>> > > un-expected
>> > > > > >> >> condition
>> > > > > >> >> > > that would leave an inconsistent state
>> > > > > >> >> > >
>> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
>> > Source
>> > > > API
>> > > > > >> and
>> > > > > >> >> > Split
>> > > > > >> >> > > Reader API
>> > > > > >> >> > >
>> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > > > > >> >> > > [refactor][tests] Move some source test utils from
>> > > > > >> >> flink-connector-base
>> > > > > >> >> > to
>> > > > > >> >> > > flink-core
>> > > > > >> >> > >
>> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
>> handling in
>> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
>> > > > > >> >> > >
>> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
>> > connector
>> > > > base
>> > > > > >> >> > > (SplitFetcherManager).
>> > > > > >> >> > >
>> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
>> > > > > >> SingleThreadFetcherManager
>> > > > > >> >> > >
>> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
>> > > > > >> >> SourceReaderBase
>> > > > > >> >> > > (part 2)
>> > > > > >> >> > >
>> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
>> > > SplitFetchers
>> > > > > more
>> > > > > >> >> > > elegantly.
>> > > > > >> >> > >
>> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > > > > >> >> > > [hotfix][tests] Extend test coverage for
>> > > > > >> FutureCompletingBlockingQueue
>> > > > > >> >> > >
>> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
>> > > > > >> >> > > FutureCompletingBlockingQueue.
>> > > > > >> >> > >
>> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
>> > Model
>> > > in
>> > > > > >> Base
>> > > > > >> >> > > Connector
>> > > > > >> >> > >
>> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers
>> lead
>> > to
>> > > > > >> >> availability
>> > > > > >> >> > > notifications.
>> > > > > >> >> > >
>> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
>> > support
>> > > > > proper
>> > > > > >> >> unit
>> > > > > >> >> > > testing
>> > > > > >> >> > >
>> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
>> relevant
>> > > to
>> > > > > only
>> > > > > >> >> one
>> > > > > >> >> > > test into test method
>> > > > > >> >> > >
>> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
>> > > > > >> >> SourceReaderBase
>> > > > > >> >> > >
>> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
>> > > > > elegantly.
>> > > > > >> >> > >
>> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
>> > > > RecordsWithSplitIds
>> > > > > >> more
>> > > > > >> >> > > lightweight.
>> > > > > >> >> > >
>> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
>> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
>> > > > > >> RecordsWithSplitIds
>> > > > > >> >> to
>> > > > > >> >> > > support reuse of heavy objects.
>> > > > > >> >> > >
>> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
>> > mocks
>> > > > for
>> > > > > >> the
>> > > > > >> >> > Split
>> > > > > >> >> > > Reader API
>> > > > > >> >> > >
>> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
>> > > > > >> 'flink-connector-base'
>> > > > > >> >> > >
>> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> > > > > >> >> > > [refactor][connectors] Backport of the connector-base
>> > > exception
>> > > > > >> >> handling
>> > > > > >> >> > > from the Kafka Connector Pull Request
>> > > > > >> >> > >
>> > > > > >> >> > >
>> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
>> > > > [hidden email]>
>> > > > > >> >> wrote:
>> > > > > >> >> > >
>> > > > > >> >> > > > I would love to see this FLIP-27 source interface
>> > > improvement
>> > > > > [1]
>> > > > > >> >> made
>> > > > > >> >> > to
>> > > > > >> >> > > > 1.11.3.
>> > > > > >> >> > > >
>> > > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
>> > > > > >> >> > > >
>> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
>> > > > > >> >> > > [hidden email]>
>> > > > > >> >> > > > wrote:
>> > > > > >> >> > > >
>> > > > > >> >> > > > > Thanks for the replies so far!
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > Just to provide a brief update on the status of
>> > blockers
>> > > > for
>> > > > > >> >> 1.11.3
>> > > > > >> >> > so
>> > > > > >> >> > > > far:
>> > > > > >> >> > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> > > > > >> >> > SourceReaderBase.pollNext
>> > > > > >> >> > > > may
>> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
>> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
>> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring
>> > from
>> > > > raw
>> > > > > >> keyed
>> > > > > >> >> > > stream
>> > > > > >> >> > > > > if it isn't the writer (
>> > > > > >> >> https://github.com/apache/flink/pull/13761)
>> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
>> > > should
>> > > > be
>> > > > > >> >> > skipping
>> > > > > >> >> > > > > unwritten key groups (
>> > > > > >> https://github.com/apache/flink/pull/13772)
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > *Merged*
>> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in
>> > case
>> > > of
>> > > > > >> >> suspended
>> > > > > >> >> > > > > ZooKeeper connection
>> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > Right now as it seems, progress is mainly blocked
>> on a
>> > > > > reviewer
>> > > > > >> >> for
>> > > > > >> >> > > > > FLINK-19717.
>> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
>> > > instabilities
>> > > > > [1]
>> > > > > >> to
>> > > > > >> >> see
>> > > > > >> >> > > if
>> > > > > >> >> > > > > there are any fixes that should be applied to
>> > > > `release-1.11`.
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > This is also a reminder, that if there are other
>> > blockers
>> > > > > that
>> > > > > >> we
>> > > > > >> >> > need
>> > > > > >> >> > > to
>> > > > > >> >> > > > > be aware of, or a need to re-establish estimated
>> time
>> > for
>> > > > > >> getting
>> > > > > >> >> > fixes
>> > > > > >> >> > > > in
>> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > Cheers,
>> > > > > >> >> > > > > Gordon
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > [1]
>> > > > > >> >> > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > >
>> > > > > >> >> > >
>> > > > > >> >> >
>> > > > > >> >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
>> > > > > >> >> [hidden email]>
>> > > > > >> >> > > > > wrote:
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a
>> number
>> > > of
>> > > > > >> >> important
>> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
>> volunteering.
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > > > Cheers,
>> > > > > >> >> > > > > > Kostas
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
>> > > [hidden email]>
>> > > > > >> wrote:
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon
>> > and
>> > > > > >> Xintong
>> > > > > >> >> for
>> > > > > >> >> > > > > > > volunteering as our release managers.
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > Best Regards,
>> > > > > >> >> > > > > > > Yu
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
>> > > > > >> >> > [hidden email]>
>> > > > > >> >> > > > > > wrote:
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
>> > > > > >> >> > > > > > > > My pleasure to help with the release process.
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > Thank you~
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > Xintong Song
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
>> Rohrmann <
>> > > > > >> >> > > > [hidden email]
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > > > > > wrote:
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon.
>> > There
>> > > > are
>> > > > > >> over
>> > > > > >> >> > 100
>> > > > > >> >> > > > > issues
>> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
>> > > soonish
>> > > > > >> 1.11.3
>> > > > > >> >> > > > release.
>> > > > > >> >> > > > > > Thanks
>> > > > > >> >> > > > > > > > > for volunteering as our release managers
>> Gordon
>> > > and
>> > > > > >> >> Xintong!
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > > Cheers,
>> > > > > >> >> > > > > > > > > Till
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
>> (Gordon)
>> > > > Tai <
>> > > > > >> >> > > > > > [hidden email]
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > > wrote:
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > >> Hi,
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
>> discussion
>> > > for
>> > > > > >> >> releasing
>> > > > > >> >> > > > Flink
>> > > > > >> >> > > > > > > > 1.11.3
>> > > > > >> >> > > > > > > > >> soon.
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> It seems like we already have a few
>> pressing
>> > > > issues
>> > > > > >> that
>> > > > > >> >> > needs
>> > > > > >> >> > > > to
>> > > > > >> >> > > > > be
>> > > > > >> >> > > > > > > > >> included in a new hotfix release:
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> - Heap-based timers’ restore behaviour
>> is
>> > > > > causing a
>> > > > > >> >> > > critical
>> > > > > >> >> > > > > > recovery
>> > > > > >> >> > > > > > > > >> issue for StateFun [1] [2] [3].
>> > > > > >> >> > > > > > > > >> - There are several robustness issues
>> for
>> > the
>> > > > > >> FLIP-27
>> > > > > >> >> new
>> > > > > >> >> > > > > source
>> > > > > >> >> > > > > > API,
>> > > > > >> >> > > > > > > > >> such as [4]. We already have some users
>> > using
>> > > > the
>> > > > > >> >> FLIP-27
>> > > > > >> >> > > API
>> > > > > >> >> > > > > > with
>> > > > > >> >> > > > > > > > >> 1.11.x,
>> > > > > >> >> > > > > > > > >> so it would be important to get those
>> fixes
>> > > in
>> > > > > for
>> > > > > >> >> 1.11.x
>> > > > > >> >> > > as
>> > > > > >> >> > > > > > well.
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Apart from the issues that are already
>> marked
>> > as
>> > > > > >> blocker
>> > > > > >> >> for
>> > > > > >> >> > > > > 1.11.3
>> > > > > >> >> > > > > > in
>> > > > > >> >> > > > > > > > our
>> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this
>> thread if
>> > > > there
>> > > > > >> is
>> > > > > >> >> > > already
>> > > > > >> >> > > > > > ongoing
>> > > > > >> >> > > > > > > > >> work for other important fixes that we
>> should
>> > > try
>> > > > to
>> > > > > >> >> > include.
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
>> > > managing
>> > > > > >> this
>> > > > > >> >> > > release,
>> > > > > >> >> > > > > and
>> > > > > >> >> > > > > > > > will
>> > > > > >> >> > > > > > > > >> try to communicate the priority of pending
>> > > > blockers
>> > > > > >> over
>> > > > > >> >> the
>> > > > > >> >> > > > next
>> > > > > >> >> > > > > > few
>> > > > > >> >> > > > > > > > >> days.
>> > > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
>> > > > critical,
>> > > > > >> we’d
>> > > > > >> >> > like
>> > > > > >> >> > > to
>> > > > > >> >> > > > > aim
>> > > > > >> >> > > > > > > > >> for a *feature
>> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)*
>> > and
>> > > > > start
>> > > > > >> the
>> > > > > >> >> > > > release
>> > > > > >> >> > > > > > voting
>> > > > > >> >> > > > > > > > >> process the week after.
>> > > > > >> >> > > > > > > > >> If that is too short of a notice and you
>> might
>> > > > need
>> > > > > >> more
>> > > > > >> >> > time,
>> > > > > >> >> > > > > > please
>> > > > > >> >> > > > > > > > let
>> > > > > >> >> > > > > > > > >> us know!
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Cheers,
>> > > > > >> >> > > > > > > > >> Gordon
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> [1]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19692
>> > > > > >> >> > > > > > > > >> [2]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19741
>> > > > > >> >> > > > > > > > >> [3]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19748
>> > > > > >> >> > > > > > > > >> [4]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19717
>> > > > > >> >> > > > > > > > >> [5]
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > >
>> > > > > >> >> > >
>> > > > > >> >> >
>> > > > > >> >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > >
>> > > > > >> >> > >
>> > > > > >> >> >
>> > > > > >> >> >
>> > > > > >> >> > --
>> > > > > >> >> > Best, Jingsong Lee
>> > > > > >> >> >
>> > > > > >> >>
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>