Kezhu Wang created FLINK-14790:
----------------------------------
Summary: Custom state backend in job graph composition without depending on concrete state backend implementation
Key: FLINK-14790
URL:
https://issues.apache.org/jira/browse/FLINK-14790 Project: Flink
Issue Type: New Feature
Components: API / DataStream
Reporter: Kezhu Wang
In Flink 1.9 and before, if we want to custom state backend, we have to supply concrete state backend implementation in {{StreamExecutionEnvironment.setStateBackend}}. This implies that we have to depend on that concrete state backend implementation in job graph composition phase since we have to construct this concrete state backend. This may not be appropriate if we separate job graph composition and execution strictly, as job graph composition requires no state backend functions besides customization.
Suppose that we are going to build a declarative streaming platform that building plain json/yaml like textual object, which may contain custom state backend configuration, with attached dependent jars to job graph and submitting generated job graph to Flink, we apparently should avoid ad-hoc state backend construction due to code reusability and indeterministic state backends configured.
So I propose to state backend customization in job graph composition phase to not depend on concrete state backend implementation.
To achieve this goal, a {{Configuration}} field, say {{StreamExecutionEnvironment.stateBackendConfiguration}}, should be sufficient.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
