Re: State migration scenario's

Posted by Yun Tang on Jun 09, 2021; 8:10am
URL: http://deprecated-apache-flink-mailing-list-archive.368.s1.nabble.com/State-migration-scenario-s-tp51272p51274.html

Hi Marlo,

One of the scenarios that we're trying to improve is to add or remove one field in state serializer.
Users might add or remove one field during their schema evolution, state processor could help it with another offline job while state migration could help it once we restart the new job.

Best
Yun Tang
________________________________
From: Marlo Ploemen <[hidden email]>
Sent: Wednesday, June 9, 2021 15:57
To: [hidden email] <[hidden email]>
Subject: State migration scenario's

Hi community,

I am looking into the data migration and schema evolution process for stateful streaming jobs. Currently, there is no orchestration support for performing these job evolutions and no in-job state migration or schema evolution syntax (as this is part of the separate state processor API). I am looking for examples (e.g. Github repositories) or scenarios of stateful streaming jobs where the orchestration of their state evolution process can improve development quality.

Best, Marlo