Re: Roadmap for Execution Mode (Batch/Streaming) and interaction with Table/SQL APIs

[Aljoscha Krettek-2]

Also cc'ing [hidden email]

On 2021/01/06 09:19, burkaygur wrote:
>1) How do these changes impact the Table and SQL APIs? Are they completely
>orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
>as well?

The answer here is a bit complicated. The Table API/SQL already use
similar techniques under the hood for BATCH execution. Our recent
changes just made similar functionality available for the DataStream
API. So in a way the changes are orthogonal.

However, the changes are relevant when you want to interoperate with the
DataStream and Table API. There it becomes relevant that both parts can
do STREAMING/BATCH execution well. We're not 100% there yet on this
front but we're tracking some work under FLIP-136 [1].

[1] https://cwiki.apache.org/confluence/x/0DN4CQ

>2) What is the best ticket to follow the roadmap & track the progress of
>this whole project. Specifically the parts about bootstrapping of state. I
>would love to help contribute to it.

I would say the best way to follow progress is the dev mailing list and
the the FLIP overview page [2]. That's not super intuitive and can be
hard to follow for outsiders. Sometimes, people such as myself will
write blog posts on the Flink website or private blogs that try and
shine a light on the development so it might help to follow some
relevant people from the project on Twitter, where such posts are often
announced.

Specifically about state bootstrapping, we don't have many concrete
thoughts yet. It would help if you could talk about some of the
requirements that you would have for this.

Best,
Aljoscha

[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best,
Aljoscha