Hi community,
I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure. Thanks to your reply. Best wishes, LakeShen |
I think you can modify the operator’s parallelism. It is only if you have
set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost. - Sivaprasanna On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]> wrote: > Hi community, > I have a question is that I cancel the flink task and retain the > checkpoint dir, then restore from the checkpoint dir ,can I change the > flink operator's parallelism,in my thoughts, I think I can't change the > flink operator's parallelism,but I am not sure. > Thanks to your reply. > > Best wishes, > LakeShen > |
Hi,
Yes, you can change the parallelism. One thing that you can not change is “max parallelism”. Piotrek > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]> wrote: > > I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost. > > - > Sivaprasanna > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email] <mailto:[hidden email]>> wrote: > Hi community, > I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure. > Thanks to your reply. > > Best wishes, > LakeShen |
Hi Piotr,
Does this also apply to savepoint? (meaning the max parallelism should not change for job resume from savepoint?) Thanks a lot! Eleanore On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]> wrote: > Hi, > > Yes, you can change the parallelism. One thing that you can not change is > “max parallelism”. > > Piotrek > > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]> > wrote: > > > > I think you can modify the operator’s parallelism. It is only if you > have set maxParallelism, and while restoring from a checkpoint, you > shouldn’t modify the maxParallelism. Otherwise, I believe the state will be > lost. > > > > - > > Sivaprasanna > > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email] > <mailto:[hidden email]>> wrote: > > Hi community, > > I have a question is that I cancel the flink task and retain the > checkpoint dir, then restore from the checkpoint dir ,can I change the > flink operator's parallelism,in my thoughts, I think I can't change the > flink operator's parallelism,but I am not sure. > > Thanks to your reply. > > > > Best wishes, > > LakeShen > > |
Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism . Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道: > Hi Piotr, > Does this also apply to savepoint? (meaning the max parallelism should not > change for job resume from savepoint?) > > Thanks a lot! > Eleanore > > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]> > wrote: > > > Hi, > > > > Yes, you can change the parallelism. One thing that you can not change is > > “max parallelism”. > > > > Piotrek > > > > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]> > > wrote: > > > > > > I think you can modify the operator’s parallelism. It is only if you > > have set maxParallelism, and while restoring from a checkpoint, you > > shouldn’t modify the maxParallelism. Otherwise, I believe the state will > be > > lost. > > > > > > - > > > Sivaprasanna > > > > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email] > > <mailto:[hidden email]>> wrote: > > > Hi community, > > > I have a question is that I cancel the flink task and retain the > > checkpoint dir, then restore from the checkpoint dir ,can I change the > > flink operator's parallelism,in my thoughts, I think I can't change the > > flink operator's parallelism,but I am not sure. > > > Thanks to your reply. > > > > > > Best wishes, > > > LakeShen > > > > > |
If you want to change the max parallelism then you need to take a savepoint
and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html Cheers, Till On Sat, Mar 14, 2020 at 4:07 AM LakeShen <[hidden email]> wrote: > Hi Eleanore , if you resume from savepoint , you can't change the flink > operator's max parallelism . > > Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道: > > > Hi Piotr, > > Does this also apply to savepoint? (meaning the max parallelism should > not > > change for job resume from savepoint?) > > > > Thanks a lot! > > Eleanore > > > > On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]> > > wrote: > > > > > Hi, > > > > > > Yes, you can change the parallelism. One thing that you can not change > is > > > “max parallelism”. > > > > > > Piotrek > > > > > > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]> > > > wrote: > > > > > > > > I think you can modify the operator’s parallelism. It is only if you > > > have set maxParallelism, and while restoring from a checkpoint, you > > > shouldn’t modify the maxParallelism. Otherwise, I believe the state > will > > be > > > lost. > > > > > > > > - > > > > Sivaprasanna > > > > > > > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email] > > > <mailto:[hidden email]>> wrote: > > > > Hi community, > > > > I have a question is that I cancel the flink task and retain > the > > > checkpoint dir, then restore from the checkpoint dir ,can I change the > > > flink operator's parallelism,in my thoughts, I think I can't change the > > > flink operator's parallelism,but I am not sure. > > > > Thanks to your reply. > > > > > > > > Best wishes, > > > > LakeShen > > > > > > > > > |
Free forum by Nabble | Edit this page |