Support user code as anonymous inner classes in streaming API

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Support user code as anonymous inner classes in streaming API

Hermann Gábor
Hey,

We saw that the Flink API supports user code as anonymous classes, and we
would like to support it too. So far we've been using the default Java
serialization for serializing user codes and by default that cannot
serialize (non-static) inner classes. It seems to me that UserCodeWrapper
is the workaround for this, but I could not figure it how to wrap simply
the user code (e.g. a MapFunction) and use it for serialization. Can you
show us an example or give us some hints? Thanks for the help!

Regards,
Gábor Hermann
Reply | Threaded
Open this post in threaded view
|

Re: Support user code as anonymous inner classes in streaming API

Stephan Ewen
The UserCodeWrapper is only a tool to support either setting instances of a
class, or the class itself. The underlying system Mappers for example as
either "new MapFunction()", or as "MapFunction.class".

To support non serializable types (which anonymous classes are unless the
surrounding class is serializable), we'd need to switch the serialization
Framework for UDFs (for example to Kryo). That is actually a worthwhile
effort, if you want to tackle it. It is also a fairly isolated change.

Stephan
Reply | Threaded
Open this post in threaded view
|

Re: Support user code as anonymous inner classes in streaming API

Hermann Gábor
Okay. I'm gonna try serializing with Kryo. Thanks for the fast reply!


On Wed, Jul 9, 2014 at 10:28 AM, Stephan Ewen <[hidden email]> wrote:

> The UserCodeWrapper is only a tool to support either setting instances of a
> class, or the class itself. The underlying system Mappers for example as
> either "new MapFunction()", or as "MapFunction.class".
>
> To support non serializable types (which anonymous classes are unless the
> surrounding class is serializable), we'd need to switch the serialization
> Framework for UDFs (for example to Kryo). That is actually a worthwhile
> effort, if you want to tackle it. It is also a fairly isolated change.
>
> Stephan
>