Savepoint and restart in the same command?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Savepoint and restart in the same command?

Ron Crocker
I attended Jamie Grier’s excellent presentation at Flink Forward today ("Robust Stream Processing with Apache Flink” - http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/ <http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/>) where he showed recovery from a savepoint. It took at least 3 steps - create the savepoint, kill the current job, start from savepoint. Is it possible to combine these into a single action - not just from the tooling perspective (where I think it’s pretty straightforward to simulate what I’m asking for here) but to actually have the running job STOP once it processes the save point request AND start a new job from that savepoint?

What I’m looking to reduce the gap between the savepoint creation AND the stopping of the current task as well as reducing the gap between stopping the current job and starting a new job from the savepoint.

Ron

Ron Crocker
Principal Engineer & Architect
( ( •)) New Relic
[hidden email]
M: +1 630 363 8835

Reply | Threaded
Open this post in threaded view
|

Re: Savepoint and restart in the same command?

Konstantin Knauf
Hi Ron,

I think, Till mentioned in his talk on re-scaling streaming jobs, that a
"Savepoint and Stop" command is one of the upcoming features in the
"dynamic scaling" thread.

Cheers,

Konstantin

On 12.09.2016 14:39, Ron Crocker wrote:

> I attended Jamie Grier’s excellent presentation at Flink Forward today ("Robust Stream Processing with Apache Flink” - http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/ <http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/>) where he showed recovery from a savepoint. It took at least 3 steps - create the savepoint, kill the current job, start from savepoint. Is it possible to combine these into a single action - not just from the tooling perspective (where I think it’s pretty straightforward to simulate what I’m asking for here) but to actually have the running job STOP once it processes the save point request AND start a new job from that savepoint?
>
> What I’m looking to reduce the gap between the savepoint creation AND the stopping of the current task as well as reducing the gap between stopping the current job and starting a new job from the savepoint.
>
> Ron
> —
> Ron Crocker
> Principal Engineer & Architect
> ( ( •)) New Relic
> [hidden email]
> M: +1 630 363 8835
>
>
--
Konstantin Knauf * [hidden email] * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Savepoint and restart in the same command?

Till Rohrmann
Hi Ron,

Konstantin is absolutely right. One of the missing features is an atomic
stop with savepoint operation to shut down a job in a clean manner. We will
hopefully have this soon.

Cheers,
Till

On Tue, Sep 13, 2016 at 11:53 AM, Konstantin Knauf <
[hidden email]> wrote:

> Hi Ron,
>
> I think, Till mentioned in his talk on re-scaling streaming jobs, that a
> "Savepoint and Stop" command is one of the upcoming features in the
> "dynamic scaling" thread.
>
> Cheers,
>
> Konstantin
>
> On 12.09.2016 14:39, Ron Crocker wrote:
> > I attended Jamie Grier’s excellent presentation at Flink Forward today
> ("Robust Stream Processing with Apache Flink” -
> http://flink-forward.org/kb_sessions/robust-stream-
> processing-with-apache-flink/ <http://flink-forward.org/kb_
> sessions/robust-stream-processing-with-apache-flink/>) where he showed
> recovery from a savepoint. It took at least 3 steps - create the savepoint,
> kill the current job, start from savepoint. Is it possible to combine these
> into a single action - not just from the tooling perspective (where I think
> it’s pretty straightforward to simulate what I’m asking for here) but to
> actually have the running job STOP once it processes the save point request
> AND start a new job from that savepoint?
> >
> > What I’m looking to reduce the gap between the savepoint creation AND
> the stopping of the current task as well as reducing the gap between
> stopping the current job and starting a new job from the savepoint.
> >
> > Ron
> > —
> > Ron Crocker
> > Principal Engineer & Architect
> > ( ( •)) New Relic
> > [hidden email]
> > M: +1 630 363 8835
> >
> >
>
> --
> Konstantin Knauf * [hidden email] * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>