Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

LakeShen
Hi community,
      I have a question is that I cancel the flink task and retain the
checkpoint dir, then restore from the checkpoint dir ,can I change the
flink operator's parallelism,in my thoughts, I think I can't change the
flink operator's parallelism,but I am not sure.
     Thanks to your reply.

Best wishes,
LakeShen
Reply | Threaded
Open this post in threaded view
|

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

Sivaprasanna
I think you can modify the operator’s parallelism. It is only if you have
set maxParallelism, and while restoring from a checkpoint, you shouldn’t
modify the maxParallelism. Otherwise, I believe the state will be lost.

-
Sivaprasanna

On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]> wrote:

> Hi community,
>       I have a question is that I cancel the flink task and retain the
> checkpoint dir, then restore from the checkpoint dir ,can I change the
> flink operator's parallelism,in my thoughts, I think I can't change the
> flink operator's parallelism,but I am not sure.
>      Thanks to your reply.
>
> Best wishes,
> LakeShen
>
Reply | Threaded
Open this post in threaded view
|

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

Piotr Nowojski-3
Hi,

Yes, you can change the parallelism. One thing that you can not change is “max parallelism”.

Piotrek

> On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]> wrote:
>
> I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost.
>
> -
> Sivaprasanna
>
> On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email] <mailto:[hidden email]>> wrote:
> Hi community,
>       I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure.
>      Thanks to your reply.
>
> Best wishes,
> LakeShen

Reply | Threaded
Open this post in threaded view
|

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

Eleanore Jin
Hi Piotr,
Does this also apply to savepoint? (meaning the max parallelism should not
change for job resume from savepoint?)

Thanks a lot!
Eleanore

On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]> wrote:

> Hi,
>
> Yes, you can change the parallelism. One thing that you can not change is
> “max parallelism”.
>
> Piotrek
>
> > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> wrote:
> >
> > I think you can modify the operator’s parallelism. It is only if you
> have set maxParallelism, and while restoring from a checkpoint, you
> shouldn’t modify the maxParallelism. Otherwise, I believe the state will be
> lost.
> >
> > -
> > Sivaprasanna
> >
> > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> <mailto:[hidden email]>> wrote:
> > Hi community,
> >       I have a question is that I cancel the flink task and retain the
> checkpoint dir, then restore from the checkpoint dir ,can I change the
> flink operator's parallelism,in my thoughts, I think I can't change the
> flink operator's parallelism,but I am not sure.
> >      Thanks to your reply.
> >
> > Best wishes,
> > LakeShen
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

LakeShen
Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism .

Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道:

> Hi Piotr,
> Does this also apply to savepoint? (meaning the max parallelism should not
> change for job resume from savepoint?)
>
> Thanks a lot!
> Eleanore
>
> On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Yes, you can change the parallelism. One thing that you can not change is
> > “max parallelism”.
> >
> > Piotrek
> >
> > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> > wrote:
> > >
> > > I think you can modify the operator’s parallelism. It is only if you
> > have set maxParallelism, and while restoring from a checkpoint, you
> > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> be
> > lost.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> > <mailto:[hidden email]>> wrote:
> > > Hi community,
> > >       I have a question is that I cancel the flink task and retain the
> > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > flink operator's parallelism,in my thoughts, I think I can't change the
> > flink operator's parallelism,but I am not sure.
> > >      Thanks to your reply.
> > >
> > > Best wishes,
> > > LakeShen
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

Till Rohrmann
If you want to change the max parallelism then you need to take a savepoint
and use Flink's state processor API [1] to rewrite the max parallelism by
creating a new savepoint from the old one.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html

Cheers,
Till

On Sat, Mar 14, 2020 at 4:07 AM LakeShen <[hidden email]> wrote:

> Hi Eleanore , if you resume from savepoint , you can't change the flink
> operator's max parallelism .
>
> Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道:
>
> > Hi Piotr,
> > Does this also apply to savepoint? (meaning the max parallelism should
> not
> > change for job resume from savepoint?)
> >
> > Thanks a lot!
> > Eleanore
> >
> > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, you can change the parallelism. One thing that you can not change
> is
> > > “max parallelism”.
> > >
> > > Piotrek
> > >
> > > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> > > wrote:
> > > >
> > > > I think you can modify the operator’s parallelism. It is only if you
> > > have set maxParallelism, and while restoring from a checkpoint, you
> > > shouldn’t modify the maxParallelism. Otherwise, I believe the state
> will
> > be
> > > lost.
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> > > <mailto:[hidden email]>> wrote:
> > > > Hi community,
> > > >       I have a question is that I cancel the flink task and retain
> the
> > > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > > flink operator's parallelism,in my thoughts, I think I can't change the
> > > flink operator's parallelism,but I am not sure.
> > > >      Thanks to your reply.
> > > >
> > > > Best wishes,
> > > > LakeShen
> > >
> > >
> >
>