Hi there,
So far, flink job is interpreted and deployed during bootstrap phase. Once pipeline runs, it's very hard to do partial upgrade without stop execution. (like savepoint is heavy) Is there any plan to allow upload annotated jar package which hints which stream tasks implementation CAN BE partial upgraded after next checkpoint succeed without worry about backfill. Thanks, Chen |
Hi Chen,
I think in a long-term perspective it makes sense to support things like this. The next big step is dynamic scaling without stopping the execution. Partial upgrades could be addressed afterwards, but I'm not aware of any plans. Until then, I would recommend a different architecture by using connect() and stream in a new logic dynamically. This is especially interesting for ML models etc. Regards, Timo Am 10/1/17 um 3:03 AM schrieb Chen Qin: > Hi there, > > So far, flink job is interpreted and deployed during bootstrap phase. Once > pipeline runs, it's very hard to do partial upgrade without stop execution. > (like savepoint is heavy) Is there any plan to allow upload annotated jar > package which hints which stream tasks implementation CAN BE partial > upgraded after next checkpoint succeed without worry about backfill. > > > Thanks, > Chen > |
Hi Timo,
Thanks for update. Flink pipeline deployment is very ad hoc and hard to maintain as a service platform. Each team may update their pipeline frequently which was completely blackbox and managed. Idea situation would be have a repo where all code lives and mapped to running pipelines in configuration. When new code landed ( without break topology compatibility), job manager should be able notified to pick up and load new class. Is there any doc I can follow to wire userclassloader up to prototype :) Thanks, Chen On Mon, Oct 2, 2017 at 2:20 AM, Timo Walther <[hidden email]> wrote: > Hi Chen, > > I think in a long-term perspective it makes sense to support things like > this. The next big step is dynamic scaling without stopping the execution. > Partial upgrades could be addressed afterwards, but I'm not aware of any > plans. > > Until then, I would recommend a different architecture by using connect() > and stream in a new logic dynamically. This is especially interesting for > ML models etc. > > Regards, > Timo > > > Am 10/1/17 um 3:03 AM schrieb Chen Qin: > >> Hi there, >> >> So far, flink job is interpreted and deployed during bootstrap phase. Once >> pipeline runs, it's very hard to do partial upgrade without stop >> execution. >> (like savepoint is heavy) Is there any plan to allow upload annotated jar >> package which hints which stream tasks implementation CAN BE partial >> upgraded after next checkpoint succeed without worry about backfill. >> >> >> Thanks, >> Chen >> >> > |
Free forum by Nabble | Edit this page |