Bowen Li created FLINK-15208:
--------------------------------
Summary: support client to submit both an online streaming job and an offline batch job based on dynamic catalog table
Key: FLINK-15208
URL:
https://issues.apache.org/jira/browse/FLINK-15208 Project: Flink
Issue Type: New Feature
Components: Table SQL / API, Table SQL / Client
Reporter: Bowen Li
Assignee: Bowen Li
with dynamic catalog table in FLINK-15206, users can maintain a single SQL job for both their online and offline job. However, they still need to change their configurations in order to submit different jobs over time.
E.g. when users update logic of their streaming job, they need to bootstrap both a new online job and backfill offline job, let's call them sub-jobs of a job with dynamic catalog table. They would have to
1) manually change execution mode in yaml config to "streaming" and submit the streaming job
2) manually change execution mode in yaml config to "batch" and submit the batch job
we should introduce a mechanism to allow users submit all or a subset of sub-jobs all at once. In the backfill use case mentioned above, ideally users should just execute the SQL once, and Flink should spin up two jobs for our users.
Streaming platform at some big companies like Uber and Netflix are already kind of doing this for backfill use cases one way or another - some do it in UI, some do it in planning phase. Would be great to standardize this practice and provide users with ultimate simplicity.
The assumption here is that users are fully aware of the consequences of launching two/multiple jobs at the same time. E.g. they need to handle overlapped results if there's any,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
