Need to restart flink job on yarn as supervisord does

[Shivam Sharma]

Hi,

My Flink job fails due to external activity like when Kafka goes down. I
want to restart my Flink job after certain time interval.

*I need to know best practices in this. How to restart Flink job
automatically.*

Thanks

--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- ([phone]
Email:- [hidden email]
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*

[Shivam Sharma]

​I am submitting my Flink Job on Yarn(Amazon EMR)​

On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <[hidden email]>
wrote:

> Hi,
>
> My Flink job fails due to external activity like when Kafka goes down. I
> want to restart my Flink job after certain time interval.
>
> *I need to know best practices in this. How to restart Flink job
> automatically.*
>
> Thanks
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- ([phone]
> Email:- [hidden email]
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>

--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- ([phone]
Email:- [hidden email]
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*

[Ufuk Celebi-2]

Hey Shivam,

check this out:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_strategies.html

Does it answer your questions?

– Ufuk

On Tue, Dec 26, 2017 at 9:30 AM, Shivam Sharma <[hidden email]> wrote:

> I am submitting my Flink Job on Yarn(Amazon EMR)
>
> On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <[hidden email]>
> wrote:
>
>> Hi,
>>
>> My Flink job fails due to external activity like when Kafka goes down. I
>> want to restart my Flink job after certain time interval.
>>
>> *I need to know best practices in this. How to restart Flink job
>> automatically.*
>>
>> Thanks
>>
>> --
>> Shivam Sharma
>> Data Engineer @ Goibibo
>> Indian Institute Of Information Technology, Design and Manufacturing
>> Jabalpur
>> Mobile No- ([phone]
>> Email:- [hidden email]
>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>> <https://www.linkedin.com/in/28shivamsharma>*
>>
>
>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- ([phone]
> Email:- [hidden email]
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*

[Shivam Sharma]

Hi,

I am using below restart strategy

// Retry always
env.setRestartStrategy(RestartStrategies.failureRateRestart(
    Integer.MAX_VALUE, // max failures per unit
    Time.of(20, TimeUnit.MINUTES), //time interval for measuring failure rate
    Time.of(10, TimeUnit.MINUTES) // delay
))

But I am facing OutOfMemory issue.

On Wed, Dec 27, 2017 at 1:23 PM, Ufuk Celebi <[hidden email]> wrote:

> Hey Shivam,
>
> check this out:
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_
> strategies.html
>
> Does it answer your questions?
>
> – Ufuk
>
> On Tue, Dec 26, 2017 at 9:30 AM, Shivam Sharma <[hidden email]>
> wrote:
> > I am submitting my Flink Job on Yarn(Amazon EMR)
> >
> > On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <[hidden email]
> >
> > wrote:
> >
> >> Hi,
> >>
> >> My Flink job fails due to external activity like when Kafka goes down. I
> >> want to restart my Flink job after certain time interval.
> >>
> >> *I need to know best practices in this. How to restart Flink job
> >> automatically.*
> >>
> >> Thanks
> >>
> >> --
> >> Shivam Sharma
> >> Data Engineer @ Goibibo
> >> Indian Institute Of Information Technology, Design and Manufacturing
> >> Jabalpur
> >> Mobile No- ([phone]
> >> Email:- [hidden email]
> >> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> >> <https://www.linkedin.com/in/28shivamsharma>*
> >>
> >
> >
> >
> > --
> > Shivam Sharma
> > Data Engineer @ Goibibo
> > Indian Institute Of Information Technology, Design and Manufacturing
> > Jabalpur
> > Mobile No- ([phone]
> > Email:- [hidden email]
> > LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > <https://www.linkedin.com/in/28shivamsharma>*
>

--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- ([phone]
Email:- [hidden email]
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*

[Till Rohrmann]

Hi Shivam,

could you elaborate a little bit on the OutOfMemory issue you're observing?
Maybe you could provide the logs.

Cheers,
Till

On Tue, Jan 2, 2018 at 2:12 PM, Shivam Sharma <[hidden email]>
wrote:

> Hi,
>
> I am using below restart strategy
>
> // Retry always
> env.setRestartStrategy(RestartStrategies.failureRateRestart(
>     Integer.MAX_VALUE, // max failures per unit
>     Time.of(20, TimeUnit.MINUTES), //time interval for measuring failure
> rate
>     Time.of(10, TimeUnit.MINUTES) // delay
> ))
>
> But I am facing OutOfMemory issue.
>
> On Wed, Dec 27, 2017 at 1:23 PM, Ufuk Celebi <[hidden email]> wrote:
>
> > Hey Shivam,
> >
> > check this out:
> > https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_
> > strategies.html
> >
> > Does it answer your questions?
> >
> > – Ufuk
> >
> > On Tue, Dec 26, 2017 at 9:30 AM, Shivam Sharma <[hidden email]
> >
> > wrote:
> > > I am submitting my Flink Job on Yarn(Amazon EMR)
> > >
> > > On Tue, Dec 26, 2017 at 1:59 PM, Shivam Sharma <
> [hidden email]
> > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> My Flink job fails due to external activity like when Kafka goes
> down. I
> > >> want to restart my Flink job after certain time interval.
> > >>
> > >> *I need to know best practices in this. How to restart Flink job
> > >> automatically.*
> > >>
> > >> Thanks
> > >>
> > >> --
> > >> Shivam Sharma
> > >> Data Engineer @ Goibibo
> > >> Indian Institute Of Information Technology, Design and Manufacturing
> > >> Jabalpur
> > >> Mobile No- ([phone]
> > >> Email:- [hidden email]
> > >> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > >> <https://www.linkedin.com/in/28shivamsharma>*
> > >>
> > >
> > >
> > >
> > > --
> > > Shivam Sharma
> > > Data Engineer @ Goibibo
> > > Indian Institute Of Information Technology, Design and Manufacturing
> > > Jabalpur
> > > Mobile No- ([phone]
> > > Email:- [hidden email]
> > > LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > > <https://www.linkedin.com/in/28shivamsharma>*
> >
>
>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- ([phone]
> Email:- [hidden email]
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>