[DISCUSS] FLIP-65: New type inference for Table API UDFs

[Timo Walther-2]

Hi everyone,

I would like to propose FLIP-65 that describes how we want to deal with
data types and their inference/extraction in the Table API in the
future. I have collected many comments, shortcomings, issues from users
and trainings in last years that went into the design. It completes the
work started in FLIP-37 by upgrading the type system to DataTypes and
allowing new use cases. Some key features of this FLIP are:

- Type extraction closely coupled to the SQL standard (with
precision/scale specification and complex data types)

- Simple stuff is simple, missing information can be provided with
annotations without the need to specify everything from scratch.

- Full access to the planner's type inference which allows to create
UDFs as powerful as built-in system functions

- Unification of Scala and Java API to enable the unified
TableEnvironment.registerFunction()


The design document can be found here:

https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#

I will convert it to a wiki page after the first review comments.

Happy to hear your thoughts.

Thanks,

Timo

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System

[Jark Wu-2]

Hi Timo,

Thanks for the proposal, a big +1 to the FLIP, especially this enables the
unified `TableEnvironment.registerFunction()`.

I think the design documentation is pretty good enough, I only left some
minor comments there.

Best,
Jark

On Fri, 4 Oct 2019 at 23:54, Timo Walther <[hidden email]> wrote:

> Hi everyone,
>
> I would like to propose FLIP-65 that describes how we want to deal with
> data types and their inference/extraction in the Table API in the
> future. I have collected many comments, shortcomings, issues from users
> and trainings in last years that went into the design. It completes the
> work started in FLIP-37 by upgrading the type system to DataTypes and
> allowing new use cases. Some key features of this FLIP are:
>
> - Type extraction closely coupled to the SQL standard (with
> precision/scale specification and complex data types)
>
> - Simple stuff is simple, missing information can be provided with
> annotations without the need to specify everything from scratch.
>
> - Full access to the planner's type inference which allows to create
> UDFs as powerful as built-in system functions
>
> - Unification of Scala and Java API to enable the unified
> TableEnvironment.registerFunction()
>
>
> The design document can be found here:
>
>
> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
>
> I will convert it to a wiki page after the first review comments.
>
> Happy to hear your thoughts.
>
> Thanks,
>
> Timo
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
>
>

unsubscribe



[hidden email]
 
From: Jark Wu
Date: 2019-10-08 17:29
To: dev
Subject: Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs
Hi Timo,
 
Thanks for the proposal, a big +1 to the FLIP, especially this enables the
unified `TableEnvironment.registerFunction()`.
 
I think the design documentation is pretty good enough, I only left some
minor comments there.
 
Best,
Jark
 
On Fri, 4 Oct 2019 at 23:54, Timo Walther <[hidden email]> wrote:
 

> Hi everyone,
>
> I would like to propose FLIP-65 that describes how we want to deal with
> data types and their inference/extraction in the Table API in the
> future. I have collected many comments, shortcomings, issues from users
> and trainings in last years that went into the design. It completes the
> work started in FLIP-37 by upgrading the type system to DataTypes and
> allowing new use cases. Some key features of this FLIP are:
>
> - Type extraction closely coupled to the SQL standard (with
> precision/scale specification and complex data types)
>
> - Simple stuff is simple, missing information can be provided with
> annotations without the need to specify everything from scratch.
>
> - Full access to the planner's type inference which allows to create
> UDFs as powerful as built-in system functions
>
> - Unification of Scala and Java API to enable the unified
> TableEnvironment.registerFunction()
>
>
> The design document can be found here:
>
>
> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
>
> I will convert it to a wiki page after the first review comments.
>
> Happy to hear your thoughts.
>
> Thanks,
>
> Timo
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
>
>

[Jingsong Li]

Thanks Timo for your pretty nice proposal, big +1 to the FLIP. Left some
minor comments.

A minor concern about flink-planner, precision things maybe cannot be
supported.

Best,
Jingsong Lee

On Tue, Oct 8, 2019 at 5:58 PM [hidden email] <
[hidden email]> wrote:

> unsubscribe
>
>
>
> [hidden email]
>
> From: Jark Wu
> Date: 2019-10-08 17:29
> To: dev
> Subject: Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs
> Hi Timo,
>
> Thanks for the proposal, a big +1 to the FLIP, especially this enables the
> unified `TableEnvironment.registerFunction()`.
>
> I think the design documentation is pretty good enough, I only left some
> minor comments there.
>
> Best,
> Jark
>
> On Fri, 4 Oct 2019 at 23:54, Timo Walther <[hidden email]> wrote:
>
> > Hi everyone,
> >
> > I would like to propose FLIP-65 that describes how we want to deal with
> > data types and their inference/extraction in the Table API in the
> > future. I have collected many comments, shortcomings, issues from users
> > and trainings in last years that went into the design. It completes the
> > work started in FLIP-37 by upgrading the type system to DataTypes and
> > allowing new use cases. Some key features of this FLIP are:
> >
> > - Type extraction closely coupled to the SQL standard (with
> > precision/scale specification and complex data types)
> >
> > - Simple stuff is simple, missing information can be provided with
> > annotations without the need to specify everything from scratch.
> >
> > - Full access to the planner's type inference which allows to create
> > UDFs as powerful as built-in system functions
> >
> > - Unification of Scala and Java API to enable the unified
> > TableEnvironment.registerFunction()
> >
> >
> > The design document can be found here:
> >
> >
> >
> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
> >
> > I will convert it to a wiki page after the first review comments.
> >
> > Happy to hear your thoughts.
> >
> > Thanks,
> >
> > Timo
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> >
> >
>

--
Best, Jingsong Lee

[Timo Walther-2]

Hi everyone,

thanks for the comments in the doc. The feedback so far was very
positive. I will convert the doc into a FLIP wiki page now.

If there are no further concerns, I would like to start a voting process
soon.

Thanks,
Timo

On 09.10.19 04:58, Jingsong Li wrote:

> Thanks Timo for your pretty nice proposal, big +1 to the FLIP. Left some
> minor comments.
>
> A minor concern about flink-planner, precision things maybe cannot be
> supported.
>
> Best,
> Jingsong Lee
>
> On Tue, Oct 8, 2019 at 5:58 PM [hidden email] <
> [hidden email]> wrote:
>
>> unsubscribe
>>
>>
>>
>> [hidden email]
>>
>> From: Jark Wu
>> Date: 2019-10-08 17:29
>> To: dev
>> Subject: Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs
>> Hi Timo,
>>
>> Thanks for the proposal, a big +1 to the FLIP, especially this enables the
>> unified `TableEnvironment.registerFunction()`.
>>
>> I think the design documentation is pretty good enough, I only left some
>> minor comments there.
>>
>> Best,
>> Jark
>>
>> On Fri, 4 Oct 2019 at 23:54, Timo Walther <[hidden email]> wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to propose FLIP-65 that describes how we want to deal with
>>> data types and their inference/extraction in the Table API in the
>>> future. I have collected many comments, shortcomings, issues from users
>>> and trainings in last years that went into the design. It completes the
>>> work started in FLIP-37 by upgrading the type system to DataTypes and
>>> allowing new use cases. Some key features of this FLIP are:
>>>
>>> - Type extraction closely coupled to the SQL standard (with
>>> precision/scale specification and complex data types)
>>>
>>> - Simple stuff is simple, missing information can be provided with
>>> annotations without the need to specify everything from scratch.
>>>
>>> - Full access to the planner's type inference which allows to create
>>> UDFs as powerful as built-in system functions
>>>
>>> - Unification of Scala and Java API to enable the unified
>>> TableEnvironment.registerFunction()
>>>
>>>
>>> The design document can be found here:
>>>
>>>
>>>
>> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
>>> I will convert it to a wiki page after the first review comments.
>>>
>>> Happy to hear your thoughts.
>>>
>>> Thanks,
>>>
>>> Timo
>>>
>>> [1]
>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
>>>
>

[Hequn Cheng]

+1 to start a vote.

Good to see that it is backward compatibility, as python UDFs will also be
affected.

BTW, do we target this Flip to release-1.10?

Best, Hequn

On Wed, Oct 16, 2019 at 6:19 PM Timo Walther <[hidden email]> wrote:

> Hi everyone,
>
> thanks for the comments in the doc. The feedback so far was very
> positive. I will convert the doc into a FLIP wiki page now.
>
> If there are no further concerns, I would like to start a voting process
> soon.
>
> Thanks,
> Timo
>
> On 09.10.19 04:58, Jingsong Li wrote:
> > Thanks Timo for your pretty nice proposal, big +1 to the FLIP. Left some
> > minor comments.
> >
> > A minor concern about flink-planner, precision things maybe cannot be
> > supported.
> >
> > Best,
> > Jingsong Lee
> >
> > On Tue, Oct 8, 2019 at 5:58 PM [hidden email] <
> > [hidden email]> wrote:
> >
> >> unsubscribe
> >>
> >>
> >>
> >> [hidden email]
> >>
> >> From: Jark Wu
> >> Date: 2019-10-08 17:29
> >> To: dev
> >> Subject: Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs
> >> Hi Timo,
> >>
> >> Thanks for the proposal, a big +1 to the FLIP, especially this enables
> the
> >> unified `TableEnvironment.registerFunction()`.
> >>
> >> I think the design documentation is pretty good enough, I only left some
> >> minor comments there.
> >>
> >> Best,
> >> Jark
> >>
> >> On Fri, 4 Oct 2019 at 23:54, Timo Walther <[hidden email]> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> I would like to propose FLIP-65 that describes how we want to deal with
> >>> data types and their inference/extraction in the Table API in the
> >>> future. I have collected many comments, shortcomings, issues from users
> >>> and trainings in last years that went into the design. It completes the
> >>> work started in FLIP-37 by upgrading the type system to DataTypes and
> >>> allowing new use cases. Some key features of this FLIP are:
> >>>
> >>> - Type extraction closely coupled to the SQL standard (with
> >>> precision/scale specification and complex data types)
> >>>
> >>> - Simple stuff is simple, missing information can be provided with
> >>> annotations without the need to specify everything from scratch.
> >>>
> >>> - Full access to the planner's type inference which allows to create
> >>> UDFs as powerful as built-in system functions
> >>>
> >>> - Unification of Scala and Java API to enable the unified
> >>> TableEnvironment.registerFunction()
> >>>
> >>>
> >>> The design document can be found here:
> >>>
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
> >>> I will convert it to a wiki page after the first review comments.
> >>>
> >>> Happy to hear your thoughts.
> >>>
> >>> Thanks,
> >>>
> >>> Timo
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> >>>
> >
>
>

[Timo Walther-2]

Hi Hequn,

yes we should target this feature for the 1.10 release. It will enable
not only a function DDL but also UDFs in the unified `TableEnvironment`
for the Blink planner. It will also make the Hive UDF implementation
cleaner.

Regards,
Timo

On 18.10.19 05:33, Hequn Cheng wrote:

> +1 to start a vote.
>
> Good to see that it is backward compatibility, as python UDFs will also be
> affected.
>
> BTW, do we target this Flip to release-1.10?
>
> Best, Hequn
>
> On Wed, Oct 16, 2019 at 6:19 PM Timo Walther <[hidden email]> wrote:
>
>> Hi everyone,
>>
>> thanks for the comments in the doc. The feedback so far was very
>> positive. I will convert the doc into a FLIP wiki page now.
>>
>> If there are no further concerns, I would like to start a voting process
>> soon.
>>
>> Thanks,
>> Timo
>>
>> On 09.10.19 04:58, Jingsong Li wrote:
>>> Thanks Timo for your pretty nice proposal, big +1 to the FLIP. Left some
>>> minor comments.
>>>
>>> A minor concern about flink-planner, precision things maybe cannot be
>>> supported.
>>>
>>> Best,
>>> Jingsong Lee
>>>
>>> On Tue, Oct 8, 2019 at 5:58 PM [hidden email] <
>>> [hidden email]> wrote:
>>>
>>>> unsubscribe
>>>>
>>>>
>>>>
>>>> [hidden email]
>>>>
>>>> From: Jark Wu
>>>> Date: 2019-10-08 17:29
>>>> To: dev
>>>> Subject: Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs
>>>> Hi Timo,
>>>>
>>>> Thanks for the proposal, a big +1 to the FLIP, especially this enables
>> the
>>>> unified `TableEnvironment.registerFunction()`.
>>>>
>>>> I think the design documentation is pretty good enough, I only left some
>>>> minor comments there.
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>> On Fri, 4 Oct 2019 at 23:54, Timo Walther <[hidden email]> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I would like to propose FLIP-65 that describes how we want to deal with
>>>>> data types and their inference/extraction in the Table API in the
>>>>> future. I have collected many comments, shortcomings, issues from users
>>>>> and trainings in last years that went into the design. It completes the
>>>>> work started in FLIP-37 by upgrading the type system to DataTypes and
>>>>> allowing new use cases. Some key features of this FLIP are:
>>>>>
>>>>> - Type extraction closely coupled to the SQL standard (with
>>>>> precision/scale specification and complex data types)
>>>>>
>>>>> - Simple stuff is simple, missing information can be provided with
>>>>> annotations without the need to specify everything from scratch.
>>>>>
>>>>> - Full access to the planner's type inference which allows to create
>>>>> UDFs as powerful as built-in system functions
>>>>>
>>>>> - Unification of Scala and Java API to enable the unified
>>>>> TableEnvironment.registerFunction()
>>>>>
>>>>>
>>>>> The design document can be found here:
>>>>>
>>>>>
>>>>>
>> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
>>>>> I will convert it to a wiki page after the first review comments.
>>>>>
>>>>> Happy to hear your thoughts.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Timo
>>>>>
>>>>> [1]
>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
>>