Support InMemoryTableSink or support retraction in CsvTableSink

[Fan X]

Hi everyone,

Recently I am using Flink SQL API to test some SQL queries. I run a small
dataset and output the result to CsvTableSink so that I can easily open the
file and verify the result by eye. But unfortunately, I found that
CsvTableSink does not support retraction message and thus I could not
output some of the query results to CSV files and verify them.

So I am wondering can we either support an InMemoryTableSink or support
retraction in CsvTableSink so that Flink developers can easily debug and
verify their result in a small dataset without introducing an extra system
such as HBase or MySQL.

Thanks.

[xingcanc]

Hi Fan,

It’s not hard to implement an “InMemoryTableSink” for debugging use by yourself.

The main part should be a SinkFunction that deals with Tuple<Boolean, T> records. You can either show the changelog (the record itself) or print the materialized result (e.g., a result list) for each received item.

See [1][2] for an example.

Best,
Xingcan

[1] https://github.com/apache/flink/blob/84eec21108f2c05fa872c9a3735457d73f75dc51/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/runtime/stream/table/TableSinkITCase.scala#L647 <https://github.com/apache/flink/blob/84eec21108f2c05fa872c9a3735457d73f75dc51/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/runtime/stream/table/TableSinkITCase.scala#L647>
[2] https://github.com/apache/flink/blob/84eec21108f2c05fa872c9a3735457d73f75dc51/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/runtime/stream/table/TableSinkITCase.scala#L717 <https://github.com/apache/flink/blob/84eec21108f2c05fa872c9a3735457d73f75dc51/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/runtime/stream/table/TableSinkITCase.scala#L717>

> On Jul 16, 2019, at 1:35 AM, Fan X <[hidden email]> wrote:
>
> Hi everyone,
>
> Recently I am using Flink SQL API to test some SQL queries. I run a small
> dataset and output the result to CsvTableSink so that I can easily open the
> file and verify the result by eye. But unfortunately, I found that
> CsvTableSink does not support retraction message and thus I could not
> output some of the query results to CSV files and verify them.
>
> So I am wondering can we either support an InMemoryTableSink or support
> retraction in CsvTableSink so that Flink developers can easily debug and
> verify their result in a small dataset without introducing an extra system
> such as HBase or MySQL.
>
> Thanks.