Bowen Li created FLINK-15985:
--------------------------------
Summary: offload runtime params from DDL to table hints in DML/queries
Key: FLINK-15985
URL:
https://issues.apache.org/jira/browse/FLINK-15985 Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Reporter: Bowen Li
Assignee: Danny Chen
Fix For: 1.11.0
background:
Currently Flink DDL mixes three types of params all together:
* External data’s metadata: defines what the data looks like (schema), where it is (location/url), how it should be accessed (username/pwd)
* Source/sink runtime params: defines how and usually how fast Flink source/sink reads/writes data, not affecting the results
* Kafka “sink-partitioner”
* Elastic “bulk-flush.interval/max-size/...”
* Semantics params: defines aspects like how much data Flink reads/writes, how the result will look like
* Kafka “startup-mode”, “offset”
* Watermark, timestamp column
Problems of the current mix-up: Flink cannot leverage catalogs and external system metadata alone to run queries with all the non-metadata params involved in DDL. E.g. when we add a catalog for Confluent Schema Registry, the expected user experience should be that Flink users just configure the catalog with url and usr/pwd, and should be able to run queries immediately; however, that’s not the case right now because users still have to use DDL to define a bunch params like “startup-mode”, “offset”, timestamp column, etc, along with the schema redundantly. We’ve heard many user complaints on this.
cc [~ykt836] [~lirui] [~lzljs3620320] [~jark] [~twalthr] [~dwysakowicz]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
