I attended Jamie Grier’s excellent presentation at Flink Forward today ("Robust Stream Processing with Apache Flink” - http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/ <http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/>) where he showed recovery from a savepoint. It took at least 3 steps - create the savepoint, kill the current job, start from savepoint. Is it possible to combine these into a single action - not just from the tooling perspective (where I think it’s pretty straightforward to simulate what I’m asking for here) but to actually have the running job STOP once it processes the save point request AND start a new job from that savepoint?
What I’m looking to reduce the gap between the savepoint creation AND the stopping of the current task as well as reducing the gap between stopping the current job and starting a new job from the savepoint. Ron — Ron Crocker Principal Engineer & Architect ( ( •)) New Relic [hidden email] M: +1 630 363 8835 |
Hi Ron,
I think, Till mentioned in his talk on re-scaling streaming jobs, that a "Savepoint and Stop" command is one of the upcoming features in the "dynamic scaling" thread. Cheers, Konstantin On 12.09.2016 14:39, Ron Crocker wrote: > I attended Jamie Grier’s excellent presentation at Flink Forward today ("Robust Stream Processing with Apache Flink” - http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/ <http://flink-forward.org/kb_sessions/robust-stream-processing-with-apache-flink/>) where he showed recovery from a savepoint. It took at least 3 steps - create the savepoint, kill the current job, start from savepoint. Is it possible to combine these into a single action - not just from the tooling perspective (where I think it’s pretty straightforward to simulate what I’m asking for here) but to actually have the running job STOP once it processes the save point request AND start a new job from that savepoint? > > What I’m looking to reduce the gap between the savepoint creation AND the stopping of the current task as well as reducing the gap between stopping the current job and starting a new job from the savepoint. > > Ron > — > Ron Crocker > Principal Engineer & Architect > ( ( •)) New Relic > [hidden email] > M: +1 630 363 8835 > > Konstantin Knauf * [hidden email] * +49-174-3413182 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz: Unterföhring * Amtsgericht München * HRB 135082 |
Hi Ron,
Konstantin is absolutely right. One of the missing features is an atomic stop with savepoint operation to shut down a job in a clean manner. We will hopefully have this soon. Cheers, Till On Tue, Sep 13, 2016 at 11:53 AM, Konstantin Knauf < [hidden email]> wrote: > Hi Ron, > > I think, Till mentioned in his talk on re-scaling streaming jobs, that a > "Savepoint and Stop" command is one of the upcoming features in the > "dynamic scaling" thread. > > Cheers, > > Konstantin > > On 12.09.2016 14:39, Ron Crocker wrote: > > I attended Jamie Grier’s excellent presentation at Flink Forward today > ("Robust Stream Processing with Apache Flink” - > http://flink-forward.org/kb_sessions/robust-stream- > processing-with-apache-flink/ <http://flink-forward.org/kb_ > sessions/robust-stream-processing-with-apache-flink/>) where he showed > recovery from a savepoint. It took at least 3 steps - create the savepoint, > kill the current job, start from savepoint. Is it possible to combine these > into a single action - not just from the tooling perspective (where I think > it’s pretty straightforward to simulate what I’m asking for here) but to > actually have the running job STOP once it processes the save point request > AND start a new job from that savepoint? > > > > What I’m looking to reduce the gap between the savepoint creation AND > the stopping of the current task as well as reducing the gap between > stopping the current job and starting a new job from the savepoint. > > > > Ron > > — > > Ron Crocker > > Principal Engineer & Architect > > ( ( •)) New Relic > > [hidden email] > > M: +1 630 363 8835 > > > > > > -- > Konstantin Knauf * [hidden email] * +49-174-3413182 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > > |
Free forum by Nabble | Edit this page |