Checkpoint Vs Job Completion

[vijikarthi]

Hi,

I have a question regarding how checkpoint works when a job is getting
finished.

Let's imagine that I have a custom sink implementation which is making use
of checkpointing mechanism to commit the writes (i.e., data will be flushed
only when commit API call is invoked and the invocation of commit API call
is tied to the notify checkpoint complete callback method).

My pipeline has simple Source -> Transform -> Sink with checkpoint interval
say 10 seconds.

The source exits itself after emitting the last data that it has in the
queue. The data gets propagated to transform and sink. Now since the source
is complete, I would imagine that after the sink operation is done, the job
will also complete (success) without waiting for the next checkpoint to
trigger?

Is that a right assumption? If so, is there a way to force the job to wait
for completion till the next checkpoint is complete?

Regards
Vijay



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[vijikarthi]

It looks like the problem statement is similar to what we have in "FLIP-34:
Terminate/Suspend Job with Savepoint"?



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[Fabian Hueske-2]

Hi Vijay,

You are right, FLIP-34 is the right place to look at.
AFAIK, it is not possible to trigger a checkpoint after the sources
finished.

FLIP-34 will provide a few options how to handle such requirements.

Best, Fabian

Am Do., 2. Mai 2019 um 00:14 Uhr schrieb vijikarthi <[hidden email]>:

> It looks like the problem statement is similar to what we have in "FLIP-34:
> Terminate/Suspend Job with Savepoint"?
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>