[DISCUSS] Release flink-shaded 8.0

[Chesnay Schepler-3]

Hello,

I would like to kick off the next flink-shaded release next week. There
are 2 ongoing efforts that are blocked on this release:

  * [FLINK-13467] Java 11 support requires a bump to ASM to correctly
    handle Java 11 bytecode
  * [FLINK-11767] Reworking the typeSerializerSnapshotMigrationTestBase
    requires asm-commons to be added to flink-shaded-asm

Are there any other changes on anyone's radar that we will have to make
for 1.10? (will bumping calcite require anything, for example)

[Stephan Ewen]

Are we fine with the current Netty version, or would be want to bump it?

On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler <[hidden email]>
wrote:

> Hello,
>
> I would like to kick off the next flink-shaded release next week. There
> are 2 ongoing efforts that are blocked on this release:
>
>   * [FLINK-13467] Java 11 support requires a bump to ASM to correctly
>     handle Java 11 bytecode
>   * [FLINK-11767] Reworking the typeSerializerSnapshotMigrationTestBase
>     requires asm-commons to be added to flink-shaded-asm
>
> Are there any other changes on anyone's radar that we will have to make
> for 1.10? (will bumping calcite require anything, for example)
>
>
>

[Nico Kruber-3]

I quickly went through all the changelogs for Netty 4.1.32 (which we
currently use) to the latest Netty 4.1.39.Final. Below, you will find a
list of bug fixes and performance improvements that may affect us. Nice
changes we could benefit from, also for the Java > 8 efforts. The most
important ones fixing leaks etc are #8921, #9167, #9274, #9394, and the
various CompositeByteBuf fixes. The rest are mostly performance
improvements.

Since we are still early in the dev cycle for Flink 1.10, it would maybe
nice to update and verify that the new version works correctly. I'll
create a ticket and PR.


FYI (1): My own patches to bring dynamically-linked openSSL to more
distributions, namely SUSE and Arch, have not made it into a release yet.

FYI (2): We are currently using the latest version of netty-tcnative,
i.e. 2.0.25.


Nico

----------
Netty 4.1.33.Final
- Fix ClassCastException and native crash when using kqueue transport
(#8665)
- Provide a way to cache the internal nioBuffer of the PooledByteBuffer
to reduce GC (#8603)

Netty 4.1.34.Final
- Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs
related to GCLocker (#8921)
- Correctly monkey-patch id also in whe os / arch is used within library
name (#8913)
- Further reduce ensureAccessible() overhead (#8895)
- Support using an Executor to offload blocking / long-running tasks
when processing TLS / SSL via the SslHandler (#8847)
- Minimize memory footprint for AbstractChannelHandlerContext for
handlers that execute in the EventExecutor (#8786)
- Fix three bugs in CompositeByteBuf (#8773)

Netty 4.1.35.Final
- Fix possible ByteBuf leak when CompositeByteBuf is resized (#8946)
- Correctly produce ssl alert when certificate validation fails on the
client-side when using native SSL implementation (#8949)

Netty 4.1.37.Final
- Don't filter out TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (#9274)
- Try to mark child channel writable again once the parent channel
becomes writable (#9254)
- Properly debounce wakeups (#9191)
- Don't read from timerfd and eventfd on each EventLoop tick (#9192)
- Correctly detect that KeyManagerFactory is not supported when using
OpenSSL 1.1.0+ (#9170)
- Fix possible unsafe sharing of internal NIO buffer in CompositeByteBuf
(#9169)
- KQueueEventLoop won't unregister active channels reusing a file
descriptor (#9149)
- Prefer direct io buffers if direct buffers pooled (#9167)

Netty 4.1.38.Final
- Prevent ByteToMessageDecoder from overreading when !isAutoRead (#9252)
- Correctly take length of ByteBufInputStream into account for
readLine() / readByte() (#9310)
- availableSharedCapacity will be slowly exhausted (#9394)
----------

On 18/08/2019 16:47, Stephan Ewen wrote:

> Are we fine with the current Netty version, or would be want to bump it?
>
> On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hello,
>
>     I would like to kick off the next flink-shaded release next week. There
>     are 2 ongoing efforts that are blocked on this release:
>
>       * [FLINK-13467] Java 11 support requires a bump to ASM to correctly
>         handle Java 11 bytecode
>       * [FLINK-11767] Reworking the typeSerializerSnapshotMigrationTestBase
>         requires asm-commons to be added to flink-shaded-asm
>
>     Are there any other changes on anyone's radar that we will have to make
>     for 1.10? (will bumping calcite require anything, for example)
>
>

--
Nico Kruber | Solutions Architect

Follow us @VervericaData Ververica
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

[Chesnay Schepler-3]

Nico has opened a PR for bumping netty; we plan to have this merged by
tomorrow.

Unless anyone has concerns I will kick off the release on Friday.

On 19/08/2019 12:11, Nico Kruber wrote:

> I quickly went through all the changelogs for Netty 4.1.32 (which we
> currently use) to the latest Netty 4.1.39.Final. Below, you will find a
> list of bug fixes and performance improvements that may affect us. Nice
> changes we could benefit from, also for the Java > 8 efforts. The most
> important ones fixing leaks etc are #8921, #9167, #9274, #9394, and the
> various CompositeByteBuf fixes. The rest are mostly performance
> improvements.
>
> Since we are still early in the dev cycle for Flink 1.10, it would maybe
> nice to update and verify that the new version works correctly. I'll
> create a ticket and PR.
>
>
> FYI (1): My own patches to bring dynamically-linked openSSL to more
> distributions, namely SUSE and Arch, have not made it into a release yet.
>
> FYI (2): We are currently using the latest version of netty-tcnative,
> i.e. 2.0.25.
>
>
> Nico
>
> ----------
> Netty 4.1.33.Final
> - Fix ClassCastException and native crash when using kqueue transport
> (#8665)
> - Provide a way to cache the internal nioBuffer of the PooledByteBuffer
> to reduce GC (#8603)
>
> Netty 4.1.34.Final
> - Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs
> related to GCLocker (#8921)
> - Correctly monkey-patch id also in whe os / arch is used within library
> name (#8913)
> - Further reduce ensureAccessible() overhead (#8895)
> - Support using an Executor to offload blocking / long-running tasks
> when processing TLS / SSL via the SslHandler (#8847)
> - Minimize memory footprint for AbstractChannelHandlerContext for
> handlers that execute in the EventExecutor (#8786)
> - Fix three bugs in CompositeByteBuf (#8773)
>
> Netty 4.1.35.Final
> - Fix possible ByteBuf leak when CompositeByteBuf is resized (#8946)
> - Correctly produce ssl alert when certificate validation fails on the
> client-side when using native SSL implementation (#8949)
>
> Netty 4.1.37.Final
> - Don't filter out TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (#9274)
> - Try to mark child channel writable again once the parent channel
> becomes writable (#9254)
> - Properly debounce wakeups (#9191)
> - Don't read from timerfd and eventfd on each EventLoop tick (#9192)
> - Correctly detect that KeyManagerFactory is not supported when using
> OpenSSL 1.1.0+ (#9170)
> - Fix possible unsafe sharing of internal NIO buffer in CompositeByteBuf
> (#9169)
> - KQueueEventLoop won't unregister active channels reusing a file
> descriptor (#9149)
> - Prefer direct io buffers if direct buffers pooled (#9167)
>
> Netty 4.1.38.Final
> - Prevent ByteToMessageDecoder from overreading when !isAutoRead (#9252)
> - Correctly take length of ByteBufInputStream into account for
> readLine() / readByte() (#9310)
> - availableSharedCapacity will be slowly exhausted (#9394)
> ----------
>
> On 18/08/2019 16:47, Stephan Ewen wrote:
>> Are we fine with the current Netty version, or would be want to bump it?
>>
>> On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>      Hello,
>>
>>      I would like to kick off the next flink-shaded release next week. There
>>      are 2 ongoing efforts that are blocked on this release:
>>
>>        * [FLINK-13467] Java 11 support requires a bump to ASM to correctly
>>          handle Java 11 bytecode
>>        * [FLINK-11767] Reworking the typeSerializerSnapshotMigrationTestBase
>>          requires asm-commons to be added to flink-shaded-asm
>>
>>      Are there any other changes on anyone's radar that we will have to make
>>      for 1.10? (will bumping calcite require anything, for example)
>>
>>

[Stephan Ewen]

+1 to go ahead

at some point we may want to bump the Hadoop versions for which we build
the shaded jars, but that would be a another dedicated effort

On Wed, Aug 21, 2019 at 1:41 PM Chesnay Schepler <[hidden email]> wrote:

> Nico has opened a PR for bumping netty; we plan to have this merged by
> tomorrow.
>
> Unless anyone has concerns I will kick off the release on Friday.
>
> On 19/08/2019 12:11, Nico Kruber wrote:
> > I quickly went through all the changelogs for Netty 4.1.32 (which we
> > currently use) to the latest Netty 4.1.39.Final. Below, you will find a
> > list of bug fixes and performance improvements that may affect us. Nice
> > changes we could benefit from, also for the Java > 8 efforts. The most
> > important ones fixing leaks etc are #8921, #9167, #9274, #9394, and the
> > various CompositeByteBuf fixes. The rest are mostly performance
> > improvements.
> >
> > Since we are still early in the dev cycle for Flink 1.10, it would maybe
> > nice to update and verify that the new version works correctly. I'll
> > create a ticket and PR.
> >
> >
> > FYI (1): My own patches to bring dynamically-linked openSSL to more
> > distributions, namely SUSE and Arch, have not made it into a release yet.
> >
> > FYI (2): We are currently using the latest version of netty-tcnative,
> > i.e. 2.0.25.
> >
> >
> > Nico
> >
> > ----------
> > Netty 4.1.33.Final
> > - Fix ClassCastException and native crash when using kqueue transport
> > (#8665)
> > - Provide a way to cache the internal nioBuffer of the PooledByteBuffer
> > to reduce GC (#8603)
> >
> > Netty 4.1.34.Final
> > - Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs
> > related to GCLocker (#8921)
> > - Correctly monkey-patch id also in whe os / arch is used within library
> > name (#8913)
> > - Further reduce ensureAccessible() overhead (#8895)
> > - Support using an Executor to offload blocking / long-running tasks
> > when processing TLS / SSL via the SslHandler (#8847)
> > - Minimize memory footprint for AbstractChannelHandlerContext for
> > handlers that execute in the EventExecutor (#8786)
> > - Fix three bugs in CompositeByteBuf (#8773)
> >
> > Netty 4.1.35.Final
> > - Fix possible ByteBuf leak when CompositeByteBuf is resized (#8946)
> > - Correctly produce ssl alert when certificate validation fails on the
> > client-side when using native SSL implementation (#8949)
> >
> > Netty 4.1.37.Final
> > - Don't filter out TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (#9274)
> > - Try to mark child channel writable again once the parent channel
> > becomes writable (#9254)
> > - Properly debounce wakeups (#9191)
> > - Don't read from timerfd and eventfd on each EventLoop tick (#9192)
> > - Correctly detect that KeyManagerFactory is not supported when using
> > OpenSSL 1.1.0+ (#9170)
> > - Fix possible unsafe sharing of internal NIO buffer in CompositeByteBuf
> > (#9169)
> > - KQueueEventLoop won't unregister active channels reusing a file
> > descriptor (#9149)
> > - Prefer direct io buffers if direct buffers pooled (#9167)
> >
> > Netty 4.1.38.Final
> > - Prevent ByteToMessageDecoder from overreading when !isAutoRead (#9252)
> > - Correctly take length of ByteBufInputStream into account for
> > readLine() / readByte() (#9310)
> > - availableSharedCapacity will be slowly exhausted (#9394)
> > ----------
> >
> > On 18/08/2019 16:47, Stephan Ewen wrote:
> >> Are we fine with the current Netty version, or would be want to bump it?
> >>
> >> On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler <[hidden email]
> >> <mailto:[hidden email]>> wrote:
> >>
> >>      Hello,
> >>
> >>      I would like to kick off the next flink-shaded release next week.
> There
> >>      are 2 ongoing efforts that are blocked on this release:
> >>
> >>        * [FLINK-13467] Java 11 support requires a bump to ASM to
> correctly
> >>          handle Java 11 bytecode
> >>        * [FLINK-11767] Reworking the
> typeSerializerSnapshotMigrationTestBase
> >>          requires asm-commons to be added to flink-shaded-asm
> >>
> >>      Are there any other changes on anyone's radar that we will have to
> make
> >>      for 1.10? (will bumping calcite require anything, for example)
> >>
> >>
>
>