Why checkpointing is synchronized with data processing?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Why checkpointing is synchronized with data processing?

Renjie Liu
Hi, flink dev team:

I'm reading flink's source code and it seems that the process of saving
checkpointing data to external storage system will block data processing,
and I wonder why this has to be designed like this? Isn't is a potential
risk that may increase latency of processing?

Is the following strategy better?

   1. Save state data to local disk
   2. Resume data processing
   3. Saving state data to external system asynchronously

--
Liu, Renjie
Software Engineer, MVAD
Reply | Threaded
Open this post in threaded view
|

Re: Why checkpointing is synchronized with data processing?

Stephan Ewen
Hi!

Checkpoints do not really hold data processing. The only synchronous call
is to create a "state snapshot". Some state backends implement that
synchronously (like the FsStateBackend) some asynchronously
(RocksDBStateBackend).

It depends on the state backend how much is synchronous (blocking) vs
asynchronous. The checkpoint algorithm itself allows for asynchronous
persistence.

Stephan


On Sat, Oct 29, 2016 at 6:05 AM, Renjie Liu <[hidden email]> wrote:

> Hi, flink dev team:
>
> I'm reading flink's source code and it seems that the process of saving
> checkpointing data to external storage system will block data processing,
> and I wonder why this has to be designed like this? Isn't is a potential
> risk that may increase latency of processing?
>
> Is the following strategy better?
>
>    1. Save state data to local disk
>    2. Resume data processing
>    3. Saving state data to external system asynchronously
>
> --
> Liu, Renjie
> Software Engineer, MVAD
>