[DISCUSS] Graceful Shutdown Handling by UDFs.

[Kostas Kloudas-4]

Hi all,

In many cases, UDFs (User Defined Functions) need to be able to perform
application-specific actions when they stop in an orderly manner.
Currently, Flink's UDFs, and more specifically the RichFunction which
exposes lifecycle-related hooks, only have a close() method which is called
in any case of job termination. This includes any form of orderly
termination (STOP or End-Of-Stream) and termination due to an error.


The FLIP in [1] and the design document in [2] propose the addition of an
interface that will allow UDFs that implement it to perform application
specific logic in the case of graceful termination. These cases include
DRAIN and SUSPEND for streaming jobs (see FLIP-34), but also reaching the
End-Of-Stream for jobs with finite sources.

Let's have a lively discussion to solve this issue that has been around for
quite some time.

Cheers,
Kostas

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs

[2]
https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit?usp=sharing

[Konstantin Knauf-3]

Hi Klou,

+1 for this proposal. I am missing any mention of "cancel" in the design
document though. In my understanding we are not planning to deprecate
"cancel" completely (just cancel-with-savepoint, which is superseded by
"stop"). In any case we should consider it in the design document. It seems
to me that "cancel" should be consider an ungraceful shutdown, so that the
Job could be restarted from last (retained) checkpoint (as right now).

Cheers,

Konstantin

On Thu, Jul 4, 2019 at 3:21 PM Kostas Kloudas <[hidden email]> wrote:

> Hi all,
>
> In many cases, UDFs (User Defined Functions) need to be able to perform
> application-specific actions when they stop in an orderly manner.
> Currently, Flink's UDFs, and more specifically the RichFunction which
> exposes lifecycle-related hooks, only have a close() method which is called
> in any case of job termination. This includes any form of orderly
> termination (STOP or End-Of-Stream) and termination due to an error.
>
>
> The FLIP in [1] and the design document in [2] propose the addition of an
> interface that will allow UDFs that implement it to perform application
> specific logic in the case of graceful termination. These cases include
> DRAIN and SUSPEND for streaming jobs (see FLIP-34), but also reaching the
> End-Of-Stream for jobs with finite sources.
>
> Let's have a lively discussion to solve this issue that has been around for
> quite some time.
>
> Cheers,
> Kostas
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
>
> [2]
>
> https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit?usp=sharing
>

--

Konstantin Knauf | Solutions Architect

[phone]


Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010


--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

[Kostas Kloudas-4]

Hi Konstantin,

Yes you are right that `cancel` falls under the "abrupt" termination of a
job.
I will update the FLIP accordingly.

Cheers,
Kostas

On Sun, Jul 7, 2019 at 11:38 AM Konstantin Knauf <[hidden email]>
wrote:

> Hi Klou,
>
> +1 for this proposal. I am missing any mention of "cancel" in the design
> document though. In my understanding we are not planning to deprecate
> "cancel" completely (just cancel-with-savepoint, which is superseded by
> "stop"). In any case we should consider it in the design document. It seems
> to me that "cancel" should be consider an ungraceful shutdown, so that the
> Job could be restarted from last (retained) checkpoint (as right now).
>
> Cheers,
>
> Konstantin
>
> On Thu, Jul 4, 2019 at 3:21 PM Kostas Kloudas <[hidden email]> wrote:
>
> > Hi all,
> >
> > In many cases, UDFs (User Defined Functions) need to be able to perform
> > application-specific actions when they stop in an orderly manner.
> > Currently, Flink's UDFs, and more specifically the RichFunction which
> > exposes lifecycle-related hooks, only have a close() method which is
> called
> > in any case of job termination. This includes any form of orderly
> > termination (STOP or End-Of-Stream) and termination due to an error.
> >
> >
> > The FLIP in [1] and the design document in [2] propose the addition of an
> > interface that will allow UDFs that implement it to perform application
> > specific logic in the case of graceful termination. These cases include
> > DRAIN and SUSPEND for streaming jobs (see FLIP-34), but also reaching the
> > End-Of-Stream for jobs with finite sources.
> >
> > Let's have a lively discussion to solve this issue that has been around
> for
> > quite some time.
> >
> > Cheers,
> > Kostas
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> >
> > [2]
> >
> >
> https://docs.google.com/document/d/1SXfhmeiJfWqi2ITYgCgAoSDUv5PNq1T8Zu01nR5Ebog/edit?usp=sharing
> >
>
>
> --
>
> Konstantin Knauf | Solutions Architect
>
> [phone]
>
>
> Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010
>
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>