set job level TTL

[chenqin]

Hi there,

We have use case of allowing ad hoc user submit a Flink job (mostly SQL)
and user come back and verify results after period of time (eg 24hours)
without manual intervention. The idea is to have something like set job
level TTL configuration to 24hours. Jobmanager will honor 24hours of
instrument save point and proper shutdown of flink job.

Just want to know if we have job level TTL setting or parity features JIRA.


Best,
Chen

[Yun Tang]

Hi Chen

Currently, state TTL setting is per-state scoped, and there is no job-level TTL setting so far. Do you want similar settings in flink-conf to enable Flink SQL could also benefit from state TTL?

Best
Yun Tang

On 11/25/19, 12:16 AM, "Chen Qin" <[hidden email]> wrote:

    Hi there,
   
    We have use case of allowing ad hoc user submit a Flink job (mostly SQL)
    and user come back and verify results after period of time (eg 24hours)
    without manual intervention. The idea is to have something like set job
    level TTL configuration to 24hours. Jobmanager will honor 24hours of
    instrument save point and proper shutdown of flink job.
   
    Just want to know if we have job level TTL setting or parity features JIRA.
   
   
    Best,
    Chen
   

[chenqin]

I am a bit confused. Job level TTL is different from state TTL despite
share same name.

What we want to achieve in Job TTL is basically "run x time => save point
=> shutdown job"


On Sun, Nov 24, 2019 at 9:57 AM Yun Tang <[hidden email]> wrote:

> Hi Chen
>
> Currently, state TTL setting is per-state scoped, and there is no
> job-level TTL setting so far. Do you want similar settings in flink-conf to
> enable Flink SQL could also benefit from state TTL?
>
> Best
> Yun Tang
>
> On 11/25/19, 12:16 AM, "Chen Qin" <[hidden email]> wrote:
>
>     Hi there,
>
>     We have use case of allowing ad hoc user submit a Flink job (mostly
> SQL)
>     and user come back and verify results after period of time (eg 24hours)
>     without manual intervention. The idea is to have something like set job
>     level TTL configuration to 24hours. Jobmanager will honor 24hours of
>     instrument save point and proper shutdown of flink job.
>
>     Just want to know if we have job level TTL setting or parity features
> JIRA.
>
>
>     Best,
>     Chen
>
>
>

[Yun Tang]

Hi Chen

There existed several problems currently:
1. State TTL cannot be applied to Flink SQL
2. No  job-level state TTL
3. Window state should not be limited to TTL if existed a job-level state

If your goal is to achieve #2, we could introduce a configuration via flink-conf.yaml so that we could also achieve #1 but leave window state not limited to TTL.
I think this deserves an issue to track this request.

Best
Yun Tang

On 11/25/19, 8:04 AM, "Chen Qin" <[hidden email]> wrote:

    I am a bit confused. Job level TTL is different from state TTL despite
    share same name.
   
    What we want to achieve in Job TTL is basically "run x time => save point
    => shutdown job"
   
   
    On Sun, Nov 24, 2019 at 9:57 AM Yun Tang <[hidden email]> wrote:
   
    > Hi Chen
    >
    > Currently, state TTL setting is per-state scoped, and there is no
    > job-level TTL setting so far. Do you want similar settings in flink-conf to
    > enable Flink SQL could also benefit from state TTL?
    >
    > Best
    > Yun Tang
    >
    > On 11/25/19, 12:16 AM, "Chen Qin" <[hidden email]> wrote:
    >
    >     Hi there,
    >
    >     We have use case of allowing ad hoc user submit a Flink job (mostly
    > SQL)
    >     and user come back and verify results after period of time (eg 24hours)
    >     without manual intervention. The idea is to have something like set job
    >     level TTL configuration to 24hours. Jobmanager will honor 24hours of
    >     instrument save point and proper shutdown of flink job.
    >
    >     Just want to know if we have job level TTL setting or parity features
    > JIRA.
    >
    >
    >     Best,
    >     Chen
    >
    >
    >
   

[Andrey Zagrebin-4]

Hi Chen,

if I understand correctly, your question is not about the state with TTL,
correct me if I am wrong.

We could consider adding this feature to Flink: run x time => save point
=> shutdown job but it seems to me that it is something quite application
specific, not sure how high priority it will get in community.

Would it work for you to schedule an automated cron script to check your
running job and do those actions over CLI or REST API?

Cheers,
Andrey

On Mon, Nov 25, 2019 at 7:27 PM Yun Tang <[hidden email]> wrote:

> Hi Chen
>
> There existed several problems currently:
> 1. State TTL cannot be applied to Flink SQL
> 2. No  job-level state TTL
> 3. Window state should not be limited to TTL if existed a job-level state
>
> If your goal is to achieve #2, we could introduce a configuration via
> flink-conf.yaml so that we could also achieve #1 but leave window state not
> limited to TTL.
> I think this deserves an issue to track this request.
>
> Best
> Yun Tang
>
> On 11/25/19, 8:04 AM, "Chen Qin" <[hidden email]> wrote:
>
>     I am a bit confused. Job level TTL is different from state TTL despite
>     share same name.
>
>     What we want to achieve in Job TTL is basically "run x time => save
> point
>     => shutdown job"
>
>
>     On Sun, Nov 24, 2019 at 9:57 AM Yun Tang <[hidden email]> wrote:
>
>     > Hi Chen
>     >
>     > Currently, state TTL setting is per-state scoped, and there is no
>     > job-level TTL setting so far. Do you want similar settings in
> flink-conf to
>     > enable Flink SQL could also benefit from state TTL?
>     >
>     > Best
>     > Yun Tang
>     >
>     > On 11/25/19, 12:16 AM, "Chen Qin" <[hidden email]> wrote:
>     >
>     >     Hi there,
>     >
>     >     We have use case of allowing ad hoc user submit a Flink job
> (mostly
>     > SQL)
>     >     and user come back and verify results after period of time (eg
> 24hours)
>     >     without manual intervention. The idea is to have something like
> set job
>     >     level TTL configuration to 24hours. Jobmanager will honor
> 24hours of
>     >     instrument save point and proper shutdown of flink job.
>     >
>     >     Just want to know if we have job level TTL setting or parity
> features
>     > JIRA.
>     >
>     >
>     >     Best,
>     >     Chen
>     >
>     >
>     >
>
>
>

[chenqin]

Hi Andrey,

yes, I think flip 34 helps with this use case.

https://flink.apache.org/news/2019/08/22/release-1.9.0.html#stop-with-savepoint-flip-34

Thanks,
Chen


On Tue, Nov 26, 2019 at 7:31 AM Andrey Zagrebin <[hidden email]>
wrote:

> Hi Chen,
>
> if I understand correctly, your question is not about the state with TTL,
> correct me if I am wrong.
>
> We could consider adding this feature to Flink: run x time => save point
> => shutdown job but it seems to me that it is something quite application
> specific, not sure how high priority it will get in community.
>
> Would it work for you to schedule an automated cron script to check your
> running job and do those actions over CLI or REST API?
>
> Cheers,
> Andrey
>
> On Mon, Nov 25, 2019 at 7:27 PM Yun Tang <[hidden email]> wrote:
>
> > Hi Chen
> >
> > There existed several problems currently:
> > 1. State TTL cannot be applied to Flink SQL
> > 2. No  job-level state TTL
> > 3. Window state should not be limited to TTL if existed a job-level state
> >
> > If your goal is to achieve #2, we could introduce a configuration via
> > flink-conf.yaml so that we could also achieve #1 but leave window state
> not
> > limited to TTL.
> > I think this deserves an issue to track this request.
> >
> > Best
> > Yun Tang
> >
> > On 11/25/19, 8:04 AM, "Chen Qin" <[hidden email]> wrote:
> >
> >     I am a bit confused. Job level TTL is different from state TTL
> despite
> >     share same name.
> >
> >     What we want to achieve in Job TTL is basically "run x time => save
> > point
> >     => shutdown job"
> >
> >
> >     On Sun, Nov 24, 2019 at 9:57 AM Yun Tang <[hidden email]> wrote:
> >
> >     > Hi Chen
> >     >
> >     > Currently, state TTL setting is per-state scoped, and there is no
> >     > job-level TTL setting so far. Do you want similar settings in
> > flink-conf to
> >     > enable Flink SQL could also benefit from state TTL?
> >     >
> >     > Best
> >     > Yun Tang
> >     >
> >     > On 11/25/19, 12:16 AM, "Chen Qin" <[hidden email]> wrote:
> >     >
> >     >     Hi there,
> >     >
> >     >     We have use case of allowing ad hoc user submit a Flink job
> > (mostly
> >     > SQL)
> >     >     and user come back and verify results after period of time (eg
> > 24hours)
> >     >     without manual intervention. The idea is to have something like
> > set job
> >     >     level TTL configuration to 24hours. Jobmanager will honor
> > 24hours of
> >     >     instrument save point and proper shutdown of flink job.
> >     >
> >     >     Just want to know if we have job level TTL setting or parity
> > features
> >     > JIRA.
> >     >
> >     >
> >     >     Best,
> >     >     Chen
> >     >
> >     >
> >     >
> >
> >
> >
>