outdated version of com.esotericsoftware.kryo

[Teunissen, F.G.J. (Fred)]

Hi,

While I was resolving a dependency version conflict in our project that uses Flink v1.11.2 I noticed that Flink is using the library ‘com.esotericsoftware.kryo:kryo<https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo>:2.24.0’.
Is there a reason why this library isn’t updated to a more recent version (the last one is `com.esotericsoftware: kryo<https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)

Note. The groupId has been changed, so that could be the reason why it is missed.

Regards,
Fred Teunissen

-----------------------------------------------------------------
ATTENTION:
The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately.
-----------------------------------------------------------------

[Till Rohrmann]

Hi Fred,

one reason why this hasn't been updated is that Flink needs to ensure
backwards compatibility. Hence, if the newer Kryo library serializes
objects differently, then Flink would need to migrate old savepoints having
been stored via the old Kryo serializer to the new format. This is
currently not supported and afaik blocks upgrading this library.

Cheers,
Till

On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
<[hidden email]> wrote:

> Hi,
>
> While I was resolving a dependency version conflict in our project that
> uses Flink v1.11.2 I noticed that Flink is using the library
> ‘com.esotericsoftware.kryo:kryo<
> https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> >:2.24.0’.
> Is there a reason why this library isn’t updated to a more recent version
> (the last one is `com.esotericsoftware: kryo<
> https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
>
> Note. The groupId has been changed, so that could be the reason why it is
> missed.
>
> Regards,
> Fred Teunissen
>
> -----------------------------------------------------------------
> ATTENTION:
> The information in this e-mail is confidential and only meant for the
> intended recipient. If you are not the intended recipient, don't use or
> disclose it in any way. Please let the sender know and delete the message
> immediately.
> -----------------------------------------------------------------
>

[Niels Basjes]

Till,

So, we save the savepoints using the Kryo native format?
But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
that's ... really old !
https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0

I see that Kryo has some suggestions on the migration options:
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
One thing is that they have an extra artifact that allows having multiple
versions of Kryo in a single application to make this possible.

Niels Basjes

On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <[hidden email]> wrote:

> Hi Fred,
>
> one reason why this hasn't been updated is that Flink needs to ensure
> backwards compatibility. Hence, if the newer Kryo library serializes
> objects differently, then Flink would need to migrate old savepoints having
> been stored via the old Kryo serializer to the new format. This is
> currently not supported and afaik blocks upgrading this library.
>
> Cheers,
> Till
>
> On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> <[hidden email]> wrote:
>
> > Hi,
> >
> > While I was resolving a dependency version conflict in our project that
> > uses Flink v1.11.2 I noticed that Flink is using the library
> > ‘com.esotericsoftware.kryo:kryo<
> > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > >:2.24.0’.
> > Is there a reason why this library isn’t updated to a more recent version
> > (the last one is `com.esotericsoftware: kryo<
> > https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
> >
> > Note. The groupId has been changed, so that could be the reason why it is
> > missed.
> >
> > Regards,
> > Fred Teunissen
> >
> > -----------------------------------------------------------------
> > ATTENTION:
> > The information in this e-mail is confidential and only meant for the
> > intended recipient. If you are not the intended recipient, don't use or
> > disclose it in any way. Please let the sender know and delete the message
> > immediately.
> > -----------------------------------------------------------------
> >
>

--
Best regards / Met vriendelijke groeten,

Niels Basjes

[Till Rohrmann]

Yes, I think that Flink uses whatever key-/value-serializer is defined for
serializing the keys/values for a savepoint.

I haven't looked in detail into the problem but if Kryo offers some
migration options, then this could make the problem a bit easier.

Cheers,
Till



On Fri, Jan 15, 2021 at 2:44 PM Niels Basjes <[hidden email]> wrote:

> Till,
>
> So, we save the savepoints using the Kryo native format?
> But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
> that's ... really old !
> https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0
>
> I see that Kryo has some suggestions on the migration options:
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> One thing is that they have an extra artifact that allows having multiple
> versions of Kryo in a single application to make this possible.
>
> Niels Basjes
>
> On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi Fred,
> >
> > one reason why this hasn't been updated is that Flink needs to ensure
> > backwards compatibility. Hence, if the newer Kryo library serializes
> > objects differently, then Flink would need to migrate old savepoints
> having
> > been stored via the old Kryo serializer to the new format. This is
> > currently not supported and afaik blocks upgrading this library.
> >
> > Cheers,
> > Till
> >
> > On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> > <[hidden email]> wrote:
> >
> > > Hi,
> > >
> > > While I was resolving a dependency version conflict in our project that
> > > uses Flink v1.11.2 I noticed that Flink is using the library
> > > ‘com.esotericsoftware.kryo:kryo<
> > > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > > >:2.24.0’.
> > > Is there a reason why this library isn’t updated to a more recent
> version
> > > (the last one is `com.esotericsoftware: kryo<
> > > https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
> > >
> > > Note. The groupId has been changed, so that could be the reason why it
> is
> > > missed.
> > >
> > > Regards,
> > > Fred Teunissen
> > >
> > > -----------------------------------------------------------------
> > > ATTENTION:
> > > The information in this e-mail is confidential and only meant for the
> > > intended recipient. If you are not the intended recipient, don't use or
> > > disclose it in any way. Please let the sender know and delete the
> message
> > > immediately.
> > > -----------------------------------------------------------------
> > >
> >
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

[Seth Wiesman-4]

To expand on Till's answer, Flink uses Kryo whenever the type system does
not recognize a given type. If your job works with Kryo fallback disabled,
there should be no usage of Kryo serialization in your snapshots[1]. In
theory, it should be possible to upgrade Kryo using Flink's schema
migration capabilities but I'm not aware of anyone who has looked into the
details or is working on this.

Seth

[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/types_serialization.html#disabling-kryo-fallback

On Fri, Jan 15, 2021 at 10:46 AM Till Rohrmann <[hidden email]> wrote:

> Yes, I think that Flink uses whatever key-/value-serializer is defined for
> serializing the keys/values for a savepoint.
>
> I haven't looked in detail into the problem but if Kryo offers some
> migration options, then this could make the problem a bit easier.
>
> Cheers,
> Till
>
>
>
> On Fri, Jan 15, 2021 at 2:44 PM Niels Basjes <[hidden email]> wrote:
>
> > Till,
> >
> > So, we save the savepoints using the Kryo native format?
> > But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
> > that's ... really old !
> > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0
> >
> > I see that Kryo has some suggestions on the migration options:
> >
> >
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> > One thing is that they have an extra artifact that allows having multiple
> > versions of Kryo in a single application to make this possible.
> >
> > Niels Basjes
> >
> > On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Hi Fred,
> > >
> > > one reason why this hasn't been updated is that Flink needs to ensure
> > > backwards compatibility. Hence, if the newer Kryo library serializes
> > > objects differently, then Flink would need to migrate old savepoints
> > having
> > > been stored via the old Kryo serializer to the new format. This is
> > > currently not supported and afaik blocks upgrading this library.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> > > <[hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > While I was resolving a dependency version conflict in our project
> that
> > > > uses Flink v1.11.2 I noticed that Flink is using the library
> > > > ‘com.esotericsoftware.kryo:kryo<
> > > > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > > > >:2.24.0’.
> > > > Is there a reason why this library isn’t updated to a more recent
> > version
> > > > (the last one is `com.esotericsoftware: kryo<
> > > > https://mvnrepository.com/artifact/com.esotericsoftware/kryo
> >:5.0.3`)
> > > >
> > > > Note. The groupId has been changed, so that could be the reason why
> it
> > is
> > > > missed.
> > > >
> > > > Regards,
> > > > Fred Teunissen
> > > >
> > > > -----------------------------------------------------------------
> > > > ATTENTION:
> > > > The information in this e-mail is confidential and only meant for the
> > > > intended recipient. If you are not the intended recipient, don't use
> or
> > > > disclose it in any way. Please let the sender know and delete the
> > message
> > > > immediately.
> > > > -----------------------------------------------------------------
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>