Tzu-Li (Gordon) Tai created FLINK-13093:
-------------------------------------------
Summary: Provide an easy way to modify max parallelism using the State Processor API
Key: FLINK-13093
URL:
https://issues.apache.org/jira/browse/FLINK-13093 Project: Flink
Issue Type: Sub-task
Reporter: Tzu-Li (Gordon) Tai
Currently, the State Processor API does not easily allow one to modify the max parallelism of a job. To do that with the current state of the API, one would have to read all state that exists in a loaded {{ExistingSavepoint}}, extract them as {{DataSet}}s, and then create a {{NewSavepoint}} that has the new max parallelism with all the extracted data sets bootstrapped as new state.
It would be nice if the user could simply do something like the following (API is TBD):
{code}
ExistingSavepoint savepoint = Savepoint.load("path", env, backend);
savepoint.modifyMaxParallelism("newPath", newParallelism);
{code}
Under the hood, a batch job is launched that repartitions all existing operator state using the new max parallelism, and writes to the re-partitioned state data into the new savepoint path.
The API can be designed such that the user cannot modify the max parallelism and add / new operators at the same time, to not over complicate the batch job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
