Next steps: SQL / StreamSQL support

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Next steps: SQL / StreamSQL support

Fabian Hueske-2
Hi everybody,

on Friday we merged the working branch to put the Table API on top of
Calcite back to master.
This was the first step towards adding SQL support to Flink as outlined in
the design document [1] (the document was updated to reflect design
decisions done while implementing task 1).

According to the design doc, the next step is to add support for SQL
queries on DataSets and Table API Tables.  We created two JIRA issues to
track this effort:
- FLINK-3639: Add methods to register DataSets and Tables in
TableEnvironment
- FLINK-3640: Add support for SQL queries on registered DataSets and Tables

Subsequent efforts will be to add support for SQL queries on external
tables (CSV, Parquet, etc files, DBMS, etc.), extending coverage of SQL
standard (sort, outer joins, etc.), and defining table sinks to emit the
result.

The following document shows the syntax to register tables (DataSets,
DataStreams, Tables, external sources), query them, and to define table
sinks to write a Table to an external storage system [2].

At the same time, we are working on extending the Table API for streaming
tables (FLINK-3547).

As usual, feedback, comments, and contributions are highly welcome :-)

Best, Fabian

[1]
https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI
[2]
https://docs.google.com/document/d/1sITIShmJMGegzAjGqFuwiN_iw1urwykKsLiacokxSw0
Reply | Threaded
Open this post in threaded view
|

Re: Next steps: SQL / StreamSQL support

Vasiliki Kalavri
Thanks for the nice summary and for updating the design documents Fabian!

As we proceed with the upcoming tasks, we should also go through existing
JIRAs and update them, too.
There are some old issues referring to SQL and adding external data
sources, but these were created before the decision of using Calcite. It
would be nice to clean up theTable API JIRAs a bit by removing the invalid
issues and updating the ones that are still relevant.

Cheers,
-Vasia.

On 21 March 2016 at 17:56, Fabian Hueske <[hidden email]> wrote:

> Hi everybody,
>
> on Friday we merged the working branch to put the Table API on top of
> Calcite back to master.
> This was the first step towards adding SQL support to Flink as outlined in
> the design document [1] (the document was updated to reflect design
> decisions done while implementing task 1).
>
> According to the design doc, the next step is to add support for SQL
> queries on DataSets and Table API Tables.  We created two JIRA issues to
> track this effort:
> - FLINK-3639: Add methods to register DataSets and Tables in
> TableEnvironment
> - FLINK-3640: Add support for SQL queries on registered DataSets and Tables
>
> Subsequent efforts will be to add support for SQL queries on external
> tables (CSV, Parquet, etc files, DBMS, etc.), extending coverage of SQL
> standard (sort, outer joins, etc.), and defining table sinks to emit the
> result.
>
> The following document shows the syntax to register tables (DataSets,
> DataStreams, Tables, external sources), query them, and to define table
> sinks to write a Table to an external storage system [2].
>
> At the same time, we are working on extending the Table API for streaming
> tables (FLINK-3547).
>
> As usual, feedback, comments, and contributions are highly welcome :-)
>
> Best, Fabian
>
> [1]
>
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI
> [2]
>
> https://docs.google.com/document/d/1sITIShmJMGegzAjGqFuwiN_iw1urwykKsLiacokxSw0
>