CheckpointedFunction applicability?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

CheckpointedFunction applicability?

Eron Wright
Hello!  

The `CheckpointedFunction` interface was introduced to provide user functions with a context for partitioned state storage.   This works for user functions that are provided directly to the operator (assuming it extends from `AbstractUdfStreamOperator`).   However, I see that some operators wrap the user function in an internal function (e.g. `WrappingFunction`,  `InternalWindowFunction`).   This defeats the callbacks on `CheckpointedFunction` because those wrappers don’t implement `CheckpointedFunction`.  Is this by design or a bug?

For example, the user function passed to `WindowedStream::apply` will get callbacks for `RichFunction` but not for `CheckpointedFunction`.

To fix it, the wrapper would assumedly implement `CheckpointedFunction` and contain logic similar to that found in `AbstractUdfStreamOperator` (including the handling of `ListCheckpointed`).

Thanks
Eron
Reply | Threaded
Open this post in threaded view
|

Re: CheckpointedFunction applicability?

Stefan Richter
Hi,

you are correct, the wrappers are lacking callbacks on CheckpointedFunction. This is not the intended design and should be added.

Best,
Stefan


> Am 05.01.2017 um 04:40 schrieb Eron Wright <[hidden email]>:
>
> Hello!  
>
> The `CheckpointedFunction` interface was introduced to provide user functions with a context for partitioned state storage.   This works for user functions that are provided directly to the operator (assuming it extends from `AbstractUdfStreamOperator`).   However, I see that some operators wrap the user function in an internal function (e.g. `WrappingFunction`,  `InternalWindowFunction`).   This defeats the callbacks on `CheckpointedFunction` because those wrappers don’t implement `CheckpointedFunction`.  Is this by design or a bug?
>
> For example, the user function passed to `WindowedStream::apply` will get callbacks for `RichFunction` but not for `CheckpointedFunction`.
>
> To fix it, the wrapper would assumedly implement `CheckpointedFunction` and contain logic similar to that found in `AbstractUdfStreamOperator` (including the handling of `ListCheckpointed`).
>
> Thanks
> Eron

Reply | Threaded
Open this post in threaded view
|

Re: CheckpointedFunction applicability?

Eron Wright
Opened FLINK-5416 (unassigned).


________________________________
From: Stefan Richter <[hidden email]>
Sent: Thursday, January 5, 2017 5:54:30 AM
To: [hidden email]
Subject: Re: CheckpointedFunction applicability?

Hi,

you are correct, the wrappers are lacking callbacks on CheckpointedFunction. This is not the intended design and should be added.

Best,
Stefan


> Am 05.01.2017 um 04:40 schrieb Eron Wright <[hidden email]>:
>
> Hello!
>
> The `CheckpointedFunction` interface was introduced to provide user functions with a context for partitioned state storage.   This works for user functions that are provided directly to the operator (assuming it extends from `AbstractUdfStreamOperator`).   However, I see that some operators wrap the user function in an internal function (e.g. `WrappingFunction`,  `InternalWindowFunction`).   This defeats the callbacks on `CheckpointedFunction` because those wrappers don’t implement `CheckpointedFunction`.  Is this by design or a bug?
>
> For example, the user function passed to `WindowedStream::apply` will get callbacks for `RichFunction` but not for `CheckpointedFunction`.
>
> To fix it, the wrapper would assumedly implement `CheckpointedFunction` and contain logic similar to that found in `AbstractUdfStreamOperator` (including the handling of `ListCheckpointed`).
>
> Thanks
> Eron