[VOTE] FLIP-36 - Support Interactive Programming in Flink Table API

[Xuannan Su]

Hi all,

I'd like to start the vote for FLIP-36[1], which has been discussed in
thread[2].

The vote will be open for 72h, until May 3, 2020, 07:00 AM UTC, unless
there's an objection.

Best,
Xuannan

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td40592.html

[Timo Walther-2]

Hi Xuannan,

sorry, for not entering the discussion earlier. Could you please update
the FLIP to how it would like after FLIP-84? I think your proposal makes
sense to me and aligns well with the other efforts from an API
perspective. However, here are some thought from my side:

It would be nice to already think about how we can support this feature
in a multi-statement SQL file. Would we use planner hints (as discussed
in FLIP-113) or rather introduce some custom DCL on top of views?

Actually, the semantics of a cached table are very similar to
materialized views. Maybe we should think about unifying these concepts?

CREATE MATERIALIZED VIEW x SELECT * FROM t;

table.materialize();

table.unmaterialize();

 From a runtime perspecitive, even for streaming a user could define a
caching service that is backed by a regular table source/table sink.

Currently, people are busy with feature freeze of Flink 1.11. Maybe we
could postpone the discussion after May 15th. I guess this FLIP is
targeted for Flink 1.12 anyways.

Regards,
Timo

On 30.04.20 09:00, Xuannan Su wrote:

> Hi all,
>
> I'd like to start the vote for FLIP-36[1], which has been discussed in
> thread[2].
>
> The vote will be open for 72h, until May 3, 2020, 07:00 AM UTC, unless
> there's an objection.
>
> Best,
> Xuannan
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td40592.html
>

[Kurt Young]

+1 to what Timo has said.

One more comment about the relation of this FLIP and FLIP-84, in FLIP-84 we
start to deprecate all APIs which will buffer the table operation or plans.
You can think of APIs like `sqlUpdate`,
and `insertInto` is some kind of buffer operation, and all buffered
operations will be executed by TableEnv.execute().

This causes ambiguous API behavior since we have other operation which will
be executed eagerly, like passing
a DDL statement to `sqlUpdate`.

From the example of this FLIP, I think it still follows the buffer kind API
design which I think needs more discussion.

Best,
Kurt


On Thu, Apr 30, 2020 at 6:57 PM Timo Walther <[hidden email]> wrote:

> Hi Xuannan,
>
> sorry, for not entering the discussion earlier. Could you please update
> the FLIP to how it would like after FLIP-84? I think your proposal makes
> sense to me and aligns well with the other efforts from an API
> perspective. However, here are some thought from my side:
>
> It would be nice to already think about how we can support this feature
> in a multi-statement SQL file. Would we use planner hints (as discussed
> in FLIP-113) or rather introduce some custom DCL on top of views?
>
> Actually, the semantics of a cached table are very similar to
> materialized views. Maybe we should think about unifying these concepts?
>
> CREATE MATERIALIZED VIEW x SELECT * FROM t;
>
> table.materialize();
>
> table.unmaterialize();
>
>  From a runtime perspecitive, even for streaming a user could define a
> caching service that is backed by a regular table source/table sink.
>
> Currently, people are busy with feature freeze of Flink 1.11. Maybe we
> could postpone the discussion after May 15th. I guess this FLIP is
> targeted for Flink 1.12 anyways.
>
> Regards,
> Timo
>
> On 30.04.20 09:00, Xuannan Su wrote:
> > Hi all,
> >
> > I'd like to start the vote for FLIP-36[1], which has been discussed in
> > thread[2].
> >
> > The vote will be open for 72h, until May 3, 2020, 07:00 AM UTC, unless
> > there's an objection.
> >
> > Best,
> > Xuannan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td40592.html
> >
>
>

[Xuannan Su]

Hi,

Thanks for the feedback, @Timo @Kurt.

I agree that we can postpone the vote after May 15th since the FLIP is not
targeted for Flink 1.11 and the questions you brought up need more
discussion. I prefer moving the discussion back to the discussion thread
and keeping the vote thread clear. I will restart this voting thread after
May 15th if we come to a consensus.

I will share my thought about your feedback in the discussion thread.
Please stay tuned.

Best,
Xuannan

On Thu, Apr 30, 2020 at 9:38 PM Kurt Young <[hidden email]> wrote:

> +1 to what Timo has said.
>
> One more comment about the relation of this FLIP and FLIP-84, in FLIP-84 we
> start to deprecate all APIs which will buffer the table operation or plans.
> You can think of APIs like `sqlUpdate`,
> and `insertInto` is some kind of buffer operation, and all buffered
> operations will be executed by TableEnv.execute().
>
> This causes ambiguous API behavior since we have other operation which will
> be executed eagerly, like passing
> a DDL statement to `sqlUpdate`.
>
> From the example of this FLIP, I think it still follows the buffer kind API
> design which I think needs more discussion.
>
> Best,
> Kurt
>
>
> On Thu, Apr 30, 2020 at 6:57 PM Timo Walther <[hidden email]> wrote:
>
> > Hi Xuannan,
> >
> > sorry, for not entering the discussion earlier. Could you please update
> > the FLIP to how it would like after FLIP-84? I think your proposal makes
> > sense to me and aligns well with the other efforts from an API
> > perspective. However, here are some thought from my side:
> >
> > It would be nice to already think about how we can support this feature
> > in a multi-statement SQL file. Would we use planner hints (as discussed
> > in FLIP-113) or rather introduce some custom DCL on top of views?
> >
> > Actually, the semantics of a cached table are very similar to
> > materialized views. Maybe we should think about unifying these concepts?
> >
> > CREATE MATERIALIZED VIEW x SELECT * FROM t;
> >
> > table.materialize();
> >
> > table.unmaterialize();
> >
> >  From a runtime perspecitive, even for streaming a user could define a
> > caching service that is backed by a regular table source/table sink.
> >
> > Currently, people are busy with feature freeze of Flink 1.11. Maybe we
> > could postpone the discussion after May 15th. I guess this FLIP is
> > targeted for Flink 1.12 anyways.
> >
> > Regards,
> > Timo
> >
> > On 30.04.20 09:00, Xuannan Su wrote:
> > > Hi all,
> > >
> > > I'd like to start the vote for FLIP-36[1], which has been discussed in
> > > thread[2].
> > >
> > > The vote will be open for 72h, until May 3, 2020, 07:00 AM UTC, unless
> > > there's an objection.
> > >
> > > Best,
> > > Xuannan
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-36-Support-Interactive-Programming-in-Flink-Table-API-td40592.html
> > >
> >
> >
>