[DISCUSS] Deprecation and removal of the legacy SQL planner

[Timo Walther-2]

Hi everyone,

since Flink 1.9 we have supported two SQL planners. Most of the original
plan of FLIP-32 [1] has been implemented. The Blink code merge has been
completed and many additional features have been added exclusively to
the new planner. The new planner is now in a much better shape than the
legacy one.

In order to avoid user confusion, reduce duplicate code, and improve
maintainability and testing times of the Flink project as a whole we
would like to propose the following steps to complete FLIP-32:

In Flink 1.13:
- Deprecate the `flink-table-planner` module
- Deprecate `BatchTableEnvironment` for both Java, Scala, and Python

In Flink 1.14:
- Drop `flink-table-planner` early
- Drop many deprecated interfaces and API on demand
- Rename `flink-table-planner-blink` to `flink-table-planner`
- Rename `flink-table-runtime-blink` to `flink-table-runtime`
- Remove references of "Blink" in the code base

This will have an impact on users that still use DataSet API together
with Table API. With this change we will not support converting between
DataSet API and Table API anymore. We hope to compensate the missing
functionality in the new unified TableEnvironment and/or the batch mode
in DataStream API during 1.14 and 1.15. For this, we are looking for
further feedback which features are required in Table API/DataStream API
to have a smooth migration path.

Looking forward to your feedback.

Regards,
Timo

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions

[Kurt Young]

<cc user & user-zh mailing lists because this is a quite big API breaking
change.>

Hi Timo,

First of all I want to thank you for introducing this planner design back
in 1.9, this is a great work
that allows lots of blink features to be merged to Flink in a reasonably
short time. It greatly
accelerates the evolution speed of Table & SQL.

Everything comes with a cost, as you said, right now we are facing the
overhead of maintaining
two planners and it causes bugs and also increases imbalance between these
two planners. As
a developer and also for the good of all Table & SQL users, I also think
it's better for us to be more
focused on a single planner.

Your proposed roadmap looks good to me, +1 from my side and thanks
again for all your efforts!

Best,
Kurt


On Thu, Feb 25, 2021 at 5:01 PM Timo Walther <[hidden email]> wrote:

> Hi everyone,
>
> since Flink 1.9 we have supported two SQL planners. Most of the original
> plan of FLIP-32 [1] has been implemented. The Blink code merge has been
> completed and many additional features have been added exclusively to
> the new planner. The new planner is now in a much better shape than the
> legacy one.
>
> In order to avoid user confusion, reduce duplicate code, and improve
> maintainability and testing times of the Flink project as a whole we
> would like to propose the following steps to complete FLIP-32:
>
> In Flink 1.13:
> - Deprecate the `flink-table-planner` module
> - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
>
> In Flink 1.14:
> - Drop `flink-table-planner` early
> - Drop many deprecated interfaces and API on demand
> - Rename `flink-table-planner-blink` to `flink-table-planner`
> - Rename `flink-table-runtime-blink` to `flink-table-runtime`
> - Remove references of "Blink" in the code base
>
> This will have an impact on users that still use DataSet API together
> with Table API. With this change we will not support converting between
> DataSet API and Table API anymore. We hope to compensate the missing
> functionality in the new unified TableEnvironment and/or the batch mode
> in DataStream API during 1.14 and 1.15. For this, we are looking for
> further feedback which features are required in Table API/DataStream API
> to have a smooth migration path.
>
> Looking forward to your feedback.
>
> Regards,
> Timo
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
>

[Seth Wiesman-4]

Strong +1

Having two planners is confusing to users and the diverging semantics make
it difficult to provide useful learning material. It is time to rip the
bandage off.

Seth

On Fri, Feb 26, 2021 at 12:54 AM Kurt Young <[hidden email]> wrote:

> <cc user & user-zh mailing lists because this is a quite big API breaking
> change.>
>
> Hi Timo,
>
> First of all I want to thank you for introducing this planner design back
> in 1.9, this is a great work
> that allows lots of blink features to be merged to Flink in a reasonably
> short time. It greatly
> accelerates the evolution speed of Table & SQL.
>
> Everything comes with a cost, as you said, right now we are facing the
> overhead of maintaining
> two planners and it causes bugs and also increases imbalance between these
> two planners. As
> a developer and also for the good of all Table & SQL users, I also think
> it's better for us to be more
> focused on a single planner.
>
> Your proposed roadmap looks good to me, +1 from my side and thanks
> again for all your efforts!
>
> Best,
> Kurt
>
>
> On Thu, Feb 25, 2021 at 5:01 PM Timo Walther <[hidden email]> wrote:
>
> > Hi everyone,
> >
> > since Flink 1.9 we have supported two SQL planners. Most of the original
> > plan of FLIP-32 [1] has been implemented. The Blink code merge has been
> > completed and many additional features have been added exclusively to
> > the new planner. The new planner is now in a much better shape than the
> > legacy one.
> >
> > In order to avoid user confusion, reduce duplicate code, and improve
> > maintainability and testing times of the Flink project as a whole we
> > would like to propose the following steps to complete FLIP-32:
> >
> > In Flink 1.13:
> > - Deprecate the `flink-table-planner` module
> > - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
> >
> > In Flink 1.14:
> > - Drop `flink-table-planner` early
> > - Drop many deprecated interfaces and API on demand
> > - Rename `flink-table-planner-blink` to `flink-table-planner`
> > - Rename `flink-table-runtime-blink` to `flink-table-runtime`
> > - Remove references of "Blink" in the code base
> >
> > This will have an impact on users that still use DataSet API together
> > with Table API. With this change we will not support converting between
> > DataSet API and Table API anymore. We hope to compensate the missing
> > functionality in the new unified TableEnvironment and/or the batch mode
> > in DataStream API during 1.14 and 1.15. For this, we are looking for
> > further feedback which features are required in Table API/DataStream API
> > to have a smooth migration path.
> >
> > Looking forward to your feedback.
> >
> > Regards,
> > Timo
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> >
>

[Timo Walther-2]

Last call for feedback on this topic.

It seems everyone agrees to finally complete FLIP-32. Since FLIP-32 has
been accepted for a very long time, I think we don't need another voting
thread for executing the last implementation step. Please let me know if
you think differently.

I will start deprecating the affected classes and interfaces beginning
of next week.

Regards,
Timo


On 26.02.21 15:46, Seth Wiesman wrote:

> Strong +1
>
> Having two planners is confusing to users and the diverging semantics make
> it difficult to provide useful learning material. It is time to rip the
> bandage off.
>
> Seth
>
> On Fri, Feb 26, 2021 at 12:54 AM Kurt Young <[hidden email]> wrote:
>
>> <cc user & user-zh mailing lists because this is a quite big API breaking
>> change.>
>>
>> Hi Timo,
>>
>> First of all I want to thank you for introducing this planner design back
>> in 1.9, this is a great work
>> that allows lots of blink features to be merged to Flink in a reasonably
>> short time. It greatly
>> accelerates the evolution speed of Table & SQL.
>>
>> Everything comes with a cost, as you said, right now we are facing the
>> overhead of maintaining
>> two planners and it causes bugs and also increases imbalance between these
>> two planners. As
>> a developer and also for the good of all Table & SQL users, I also think
>> it's better for us to be more
>> focused on a single planner.
>>
>> Your proposed roadmap looks good to me, +1 from my side and thanks
>> again for all your efforts!
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Feb 25, 2021 at 5:01 PM Timo Walther <[hidden email]> wrote:
>>
>>> Hi everyone,
>>>
>>> since Flink 1.9 we have supported two SQL planners. Most of the original
>>> plan of FLIP-32 [1] has been implemented. The Blink code merge has been
>>> completed and many additional features have been added exclusively to
>>> the new planner. The new planner is now in a much better shape than the
>>> legacy one.
>>>
>>> In order to avoid user confusion, reduce duplicate code, and improve
>>> maintainability and testing times of the Flink project as a whole we
>>> would like to propose the following steps to complete FLIP-32:
>>>
>>> In Flink 1.13:
>>> - Deprecate the `flink-table-planner` module
>>> - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
>>>
>>> In Flink 1.14:
>>> - Drop `flink-table-planner` early
>>> - Drop many deprecated interfaces and API on demand
>>> - Rename `flink-table-planner-blink` to `flink-table-planner`
>>> - Rename `flink-table-runtime-blink` to `flink-table-runtime`
>>> - Remove references of "Blink" in the code base
>>>
>>> This will have an impact on users that still use DataSet API together
>>> with Table API. With this change we will not support converting between
>>> DataSet API and Table API anymore. We hope to compensate the missing
>>> functionality in the new unified TableEnvironment and/or the batch mode
>>> in DataStream API during 1.14 and 1.15. For this, we are looking for
>>> further feedback which features are required in Table API/DataStream API
>>> to have a smooth migration path.
>>>
>>> Looking forward to your feedback.
>>>
>>> Regards,
>>> Timo
>>>
>>> [1]
>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
>>>
>>
>

[Leonard Xu]

+1 for the roadmap.

Thanks Timo for driving this.

Best,
Leonard

> 在 2021年3月4日，20:40，Timo Walther <[hidden email]> 写道：
>
> Last call for feedback on this topic.
>
> It seems everyone agrees to finally complete FLIP-32. Since FLIP-32 has been accepted for a very long time, I think we don't need another voting thread for executing the last implementation step. Please let me know if you think differently.
>
> I will start deprecating the affected classes and interfaces beginning of next week.
>
> Regards,
> Timo
>
>
> On 26.02.21 15:46, Seth Wiesman wrote:
>> Strong +1
>> Having two planners is confusing to users and the diverging semantics make
>> it difficult to provide useful learning material. It is time to rip the
>> bandage off.
>> Seth
>> On Fri, Feb 26, 2021 at 12:54 AM Kurt Young <[hidden email]> wrote:
>>> <cc user & user-zh mailing lists because this is a quite big API breaking
>>> change.>
>>>
>>> Hi Timo,
>>>
>>> First of all I want to thank you for introducing this planner design back
>>> in 1.9, this is a great work
>>> that allows lots of blink features to be merged to Flink in a reasonably
>>> short time. It greatly
>>> accelerates the evolution speed of Table & SQL.
>>>
>>> Everything comes with a cost, as you said, right now we are facing the
>>> overhead of maintaining
>>> two planners and it causes bugs and also increases imbalance between these
>>> two planners. As
>>> a developer and also for the good of all Table & SQL users, I also think
>>> it's better for us to be more
>>> focused on a single planner.
>>>
>>> Your proposed roadmap looks good to me, +1 from my side and thanks
>>> again for all your efforts!
>>>
>>> Best,
>>> Kurt
>>>
>>>
>>> On Thu, Feb 25, 2021 at 5:01 PM Timo Walther <[hidden email]> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> since Flink 1.9 we have supported two SQL planners. Most of the original
>>>> plan of FLIP-32 [1] has been implemented. The Blink code merge has been
>>>> completed and many additional features have been added exclusively to
>>>> the new planner. The new planner is now in a much better shape than the
>>>> legacy one.
>>>>
>>>> In order to avoid user confusion, reduce duplicate code, and improve
>>>> maintainability and testing times of the Flink project as a whole we
>>>> would like to propose the following steps to complete FLIP-32:
>>>>
>>>> In Flink 1.13:
>>>> - Deprecate the `flink-table-planner` module
>>>> - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
>>>>
>>>> In Flink 1.14:
>>>> - Drop `flink-table-planner` early
>>>> - Drop many deprecated interfaces and API on demand
>>>> - Rename `flink-table-planner-blink` to `flink-table-planner`
>>>> - Rename `flink-table-runtime-blink` to `flink-table-runtime`
>>>> - Remove references of "Blink" in the code base
>>>>
>>>> This will have an impact on users that still use DataSet API together
>>>> with Table API. With this change we will not support converting between
>>>> DataSet API and Table API anymore. We hope to compensate the missing
>>>> functionality in the new unified TableEnvironment and/or the batch mode
>>>> in DataStream API during 1.14 and 1.15. For this, we are looking for
>>>> further feedback which features are required in Table API/DataStream API
>>>> to have a smooth migration path.
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>> [1]
>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
>>>>
>>>
>

[Jark Wu-2]

big +1 from my side.

Best,
Jark

On Thu, 4 Mar 2021 at 20:59, Leonard Xu <[hidden email]> wrote:

> +1 for the roadmap.
>
> Thanks Timo for driving this.
>
> Best,
> Leonard
>
> > 在 2021年3月4日，20:40，Timo Walther <[hidden email]> 写道：
> >
> > Last call for feedback on this topic.
> >
> > It seems everyone agrees to finally complete FLIP-32. Since FLIP-32 has
> been accepted for a very long time, I think we don't need another voting
> thread for executing the last implementation step. Please let me know if
> you think differently.
> >
> > I will start deprecating the affected classes and interfaces beginning
> of next week.
> >
> > Regards,
> > Timo
> >
> >
> > On 26.02.21 15:46, Seth Wiesman wrote:
> >> Strong +1
> >> Having two planners is confusing to users and the diverging semantics
> make
> >> it difficult to provide useful learning material. It is time to rip the
> >> bandage off.
> >> Seth
> >> On Fri, Feb 26, 2021 at 12:54 AM Kurt Young <[hidden email]> wrote:
> >>> <cc user & user-zh mailing lists because this is a quite big API
> breaking
> >>> change.>
> >>>
> >>> Hi Timo,
> >>>
> >>> First of all I want to thank you for introducing this planner design
> back
> >>> in 1.9, this is a great work
> >>> that allows lots of blink features to be merged to Flink in a
> reasonably
> >>> short time. It greatly
> >>> accelerates the evolution speed of Table & SQL.
> >>>
> >>> Everything comes with a cost, as you said, right now we are facing the
> >>> overhead of maintaining
> >>> two planners and it causes bugs and also increases imbalance between
> these
> >>> two planners. As
> >>> a developer and also for the good of all Table & SQL users, I also
> think
> >>> it's better for us to be more
> >>> focused on a single planner.
> >>>
> >>> Your proposed roadmap looks good to me, +1 from my side and thanks
> >>> again for all your efforts!
> >>>
> >>> Best,
> >>> Kurt
> >>>
> >>>
> >>> On Thu, Feb 25, 2021 at 5:01 PM Timo Walther <[hidden email]>
> wrote:
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> since Flink 1.9 we have supported two SQL planners. Most of the
> original
> >>>> plan of FLIP-32 [1] has been implemented. The Blink code merge has
> been
> >>>> completed and many additional features have been added exclusively to
> >>>> the new planner. The new planner is now in a much better shape than
> the
> >>>> legacy one.
> >>>>
> >>>> In order to avoid user confusion, reduce duplicate code, and improve
> >>>> maintainability and testing times of the Flink project as a whole we
> >>>> would like to propose the following steps to complete FLIP-32:
> >>>>
> >>>> In Flink 1.13:
> >>>> - Deprecate the `flink-table-planner` module
> >>>> - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
> >>>>
> >>>> In Flink 1.14:
> >>>> - Drop `flink-table-planner` early
> >>>> - Drop many deprecated interfaces and API on demand
> >>>> - Rename `flink-table-planner-blink` to `flink-table-planner`
> >>>> - Rename `flink-table-runtime-blink` to `flink-table-runtime`
> >>>> - Remove references of "Blink" in the code base
> >>>>
> >>>> This will have an impact on users that still use DataSet API together
> >>>> with Table API. With this change we will not support converting
> between
> >>>> DataSet API and Table API anymore. We hope to compensate the missing
> >>>> functionality in the new unified TableEnvironment and/or the batch
> mode
> >>>> in DataStream API during 1.14 and 1.15. For this, we are looking for
> >>>> further feedback which features are required in Table API/DataStream
> API
> >>>> to have a smooth migration path.
> >>>>
> >>>> Looking forward to your feedback.
> >>>>
> >>>> Regards,
> >>>> Timo
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> >>>>
> >>>
> >
>
>

[Timo Walther-2]

Thanks for the feedback everyone. You can track the progress of these
efforts in:

https://issues.apache.org/jira/browse/FLINK-21709

and

https://issues.apache.org/jira/browse/FLINK-14437

Regards,
Timo


On 04.03.21 16:08, Jark Wu wrote:

> big +1 from my side.
>
> Best,
> Jark
>
> On Thu, 4 Mar 2021 at 20:59, Leonard Xu <[hidden email]> wrote:
>
>> +1 for the roadmap.
>>
>> Thanks Timo for driving this.
>>
>> Best,
>> Leonard
>>
>>> 在 2021年3月4日，20:40，Timo Walther <[hidden email]> 写道：
>>>
>>> Last call for feedback on this topic.
>>>
>>> It seems everyone agrees to finally complete FLIP-32. Since FLIP-32 has
>> been accepted for a very long time, I think we don't need another voting
>> thread for executing the last implementation step. Please let me know if
>> you think differently.
>>>
>>> I will start deprecating the affected classes and interfaces beginning
>> of next week.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> On 26.02.21 15:46, Seth Wiesman wrote:
>>>> Strong +1
>>>> Having two planners is confusing to users and the diverging semantics
>> make
>>>> it difficult to provide useful learning material. It is time to rip the
>>>> bandage off.
>>>> Seth
>>>> On Fri, Feb 26, 2021 at 12:54 AM Kurt Young <[hidden email]> wrote:
>>>>> <cc user & user-zh mailing lists because this is a quite big API
>> breaking
>>>>> change.>
>>>>>
>>>>> Hi Timo,
>>>>>
>>>>> First of all I want to thank you for introducing this planner design
>> back
>>>>> in 1.9, this is a great work
>>>>> that allows lots of blink features to be merged to Flink in a
>> reasonably
>>>>> short time. It greatly
>>>>> accelerates the evolution speed of Table & SQL.
>>>>>
>>>>> Everything comes with a cost, as you said, right now we are facing the
>>>>> overhead of maintaining
>>>>> two planners and it causes bugs and also increases imbalance between
>> these
>>>>> two planners. As
>>>>> a developer and also for the good of all Table & SQL users, I also
>> think
>>>>> it's better for us to be more
>>>>> focused on a single planner.
>>>>>
>>>>> Your proposed roadmap looks good to me, +1 from my side and thanks
>>>>> again for all your efforts!
>>>>>
>>>>> Best,
>>>>> Kurt
>>>>>
>>>>>
>>>>> On Thu, Feb 25, 2021 at 5:01 PM Timo Walther <[hidden email]>
>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> since Flink 1.9 we have supported two SQL planners. Most of the
>> original
>>>>>> plan of FLIP-32 [1] has been implemented. The Blink code merge has
>> been
>>>>>> completed and many additional features have been added exclusively to
>>>>>> the new planner. The new planner is now in a much better shape than
>> the
>>>>>> legacy one.
>>>>>>
>>>>>> In order to avoid user confusion, reduce duplicate code, and improve
>>>>>> maintainability and testing times of the Flink project as a whole we
>>>>>> would like to propose the following steps to complete FLIP-32:
>>>>>>
>>>>>> In Flink 1.13:
>>>>>> - Deprecate the `flink-table-planner` module
>>>>>> - Deprecate `BatchTableEnvironment` for both Java, Scala, and Python
>>>>>>
>>>>>> In Flink 1.14:
>>>>>> - Drop `flink-table-planner` early
>>>>>> - Drop many deprecated interfaces and API on demand
>>>>>> - Rename `flink-table-planner-blink` to `flink-table-planner`
>>>>>> - Rename `flink-table-runtime-blink` to `flink-table-runtime`
>>>>>> - Remove references of "Blink" in the code base
>>>>>>
>>>>>> This will have an impact on users that still use DataSet API together
>>>>>> with Table API. With this change we will not support converting
>> between
>>>>>> DataSet API and Table API anymore. We hope to compensate the missing
>>>>>> functionality in the new unified TableEnvironment and/or the batch
>> mode
>>>>>> in DataStream API during 1.14 and 1.15. For this, we are looking for
>>>>>> further feedback which features are required in Table API/DataStream
>> API
>>>>>> to have a smooth migration path.
>>>>>>
>>>>>> Looking forward to your feedback.
>>>>>>
>>>>>> Regards,
>>>>>> Timo
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
>>>>>>
>>>>>
>>>
>>
>>
>