Hi all,
I‘d like to discuss a feature which supports the Flink OLAP scenario.
For OLAP scenarios, There are usually some analytical queries which running
time is relatively short. These queries are also sensitive to latency. In
the current Blink sql processing, parse/validate/optimize stages are all
need meta data from the catalog API. But each request to the catalog
requires re-run of the underlying meta query.
We may need a cached catalog which can cache the table schema and statistic
info to avoid unnecessary repeated meta requests. The most straightforward
scenario is to use Flink Batch SQL to query Hive data. If there is a Cached
Hive Catalog, we will save lots of interaction latency with HMS.
I have draft a design doc about this:
https://docs.google.com/document/d/1oL8HUpv2WaF6OkFvbH5iefXkOJB__Dal_bYsIZJA_Gk/edit?usp=sharingJira issue:
https://issues.apache.org/jira/browse/FLINK-20416IMO, this feature can further improve the stability and execution speed of
analyze query for Flink SQL.
Looking forward to your feedback, and any discussion or comments are
welcome.
--
*With kind regards
------------------------------------------------------------
Sebastian Liu 刘洋
Institute of Computing Technology, Chinese Academy of Science
Mobile\WeChat: +86—15201613655
E-mail:
[hidden email] <
[hidden email]>
QQ: 3239559*
Locked
