(DEPRECATED) Apache Flink Mailing List archive.

[VOTE] FLIP-27 - Refactor Source Interface

Classic

List

Threaded

15 messages Options

Becket Qin

[VOTE] FLIP-27 - Refactor Source Interface

Hi all,

I would like to start the vote for FLIP-27 which proposes to introduce a
new Source connector interface to address a few problems in the existing
source connector. The main goals of the the FLIP are following:

1. Unify the Source interface in Flink for batch and stream.
2. Significantly reduce the work for developers to develop new source
connectors.
3. Provide a common abstraction for all the sources, as well as a mechanism
to allow source subtasks to coordinate among themselves.

The vote will last for at least 72 hours, following the consensus voting
process.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface

Discussion thread:
https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E

Thanks,

Jiangjie (Becket) Qin

jincheng sun

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (binding), and looking forward to seeing the new interface in the master.

Best,
Jincheng

Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：

> Hi all,
>
> I would like to start the vote for FLIP-27 which proposes to introduce a
> new Source connector interface to address a few problems in the existing
> source connector. The main goals of the the FLIP are following:
>
> 1. Unify the Source interface in Flink for batch and stream.
> 2. Significantly reduce the work for developers to develop new source
> connectors.
> 3. Provide a common abstraction for all the sources, as well as a mechanism
> to allow source subtasks to coordinate among themselves.
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
>
> Discussion thread:
>
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

Becket Qin

Re: [VOTE] FLIP-27 - Refactor Source Interface

Hi Folks,

I'd like to resume the voting thread for FlIP-27.

Please note that the FLIP wiki has been updated to reflect the latest
discussions in the discussion thread.

To avoid confusion, I'll only count the votes casted after this point.

FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
%3A+Refactor+Source+Interface

Discussion thread:
https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E

The vote will last for at least 72 hours, following the consensus voting
process.

Thanks,

Jiangjie (Becket) Qin

On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <[hidden email]>
wrote:

> +1 (binding), and looking forward to seeing the new interface in the
> master.
>
> Best,
> Jincheng
>
> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
>
> > Hi all,
> >
> > I would like to start the vote for FLIP-27 which proposes to introduce a
> > new Source connector interface to address a few problems in the existing
> > source connector. The main goals of the the FLIP are following:
> >
> > 1. Unify the Source interface in Flink for batch and stream.
> > 2. Significantly reduce the work for developers to develop new source
> > connectors.
> > 3. Provide a common abstraction for all the sources, as well as a
> mechanism
> > to allow source subtasks to coordinate among themselves.
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >
> > Discussion thread:
> >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>

Becket Qin

Re: [VOTE] FLIP-27 - Refactor Source Interface

Bump up the thread.

On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]> wrote:

> Hi Folks,
>
> I'd like to resume the voting thread for FlIP-27.
>
> Please note that the FLIP wiki has been updated to reflect the latest
> discussions in the discussion thread.
>
> To avoid confusion, I'll only count the votes casted after this point.
>
> FLIP wiki:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> %3A+Refactor+Source+Interface
>
> Discussion thread:
>
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <[hidden email]>
> wrote:
>
>> +1 (binding), and looking forward to seeing the new interface in the
>> master.
>>
>> Best,
>> Jincheng
>>
>> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
>>
>> > Hi all,
>> >
>> > I would like to start the vote for FLIP-27 which proposes to introduce a
>> > new Source connector interface to address a few problems in the existing
>> > source connector. The main goals of the the FLIP are following:
>> >
>> > 1. Unify the Source interface in Flink for batch and stream.
>> > 2. Significantly reduce the work for developers to develop new source
>> > connectors.
>> > 3. Provide a common abstraction for all the sources, as well as a
>> mechanism
>> > to allow source subtasks to coordinate among themselves.
>> >
>> > The vote will last for at least 72 hours, following the consensus voting
>> > process.
>> >
>> > FLIP wiki:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
>> >
>> > Discussion thread:
>> >
>> >
>> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>>
>

Yu Li

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1, thanks for the efforts Becket!

Best Regards,
Yu

On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]> wrote:

> Bump up the thread.
>
> On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]> wrote:
>
> > Hi Folks,
> >
> > I'd like to resume the voting thread for FlIP-27.
> >
> > Please note that the FLIP wiki has been updated to reflect the latest
> > discussions in the discussion thread.
> >
> > To avoid confusion, I'll only count the votes casted after this point.
> >
> > FLIP wiki:
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > %3A+Refactor+Source+Interface
> >
> > Discussion thread:
> >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <[hidden email]>
> > wrote:
> >
> >> +1 (binding), and looking forward to seeing the new interface in the
> >> master.
> >>
> >> Best,
> >> Jincheng
> >>
> >> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> >>
> >> > Hi all,
> >> >
> >> > I would like to start the vote for FLIP-27 which proposes to
> introduce a
> >> > new Source connector interface to address a few problems in the
> existing
> >> > source connector. The main goals of the the FLIP are following:
> >> >
> >> > 1. Unify the Source interface in Flink for batch and stream.
> >> > 2. Significantly reduce the work for developers to develop new source
> >> > connectors.
> >> > 3. Provide a common abstraction for all the sources, as well as a
> >> mechanism
> >> > to allow source subtasks to coordinate among themselves.
> >> >
> >> > The vote will last for at least 72 hours, following the consensus
> voting
> >> > process.
> >> >
> >> > FLIP wiki:
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >> >
> >> > Discussion thread:
> >> >
> >> >
> >>
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> >> >
> >> > Thanks,
> >> >
> >> > Jiangjie (Becket) Qin
> >> >
> >>
> >
>

Jark Wu-2

Re: [VOTE] FLIP-27 - Refactor Source Interface

Thanks for driving this Becket!

+1 from my side.

Cheers,
Jark

On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:

> +1, thanks for the efforts Becket!
>
> Best Regards,
> Yu
>
>
> On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]> wrote:
>
> > Bump up the thread.
> >
> > On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]>
> wrote:
> >
> > > Hi Folks,
> > >
> > > I'd like to resume the voting thread for FlIP-27.
> > >
> > > Please note that the FLIP wiki has been updated to reflect the latest
> > > discussions in the discussion thread.
> > >
> > > To avoid confusion, I'll only count the votes casted after this point.
> > >
> > > FLIP wiki:
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > > %3A+Refactor+Source+Interface
> > >
> > > Discussion thread:
> > >
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <[hidden email]
> >
> > > wrote:
> > >
> > >> +1 (binding), and looking forward to seeing the new interface in the
> > >> master.
> > >>
> > >> Best,
> > >> Jincheng
> > >>
> > >> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> > >>
> > >> > Hi all,
> > >> >
> > >> > I would like to start the vote for FLIP-27 which proposes to
> > introduce a
> > >> > new Source connector interface to address a few problems in the
> > existing
> > >> > source connector. The main goals of the the FLIP are following:
> > >> >
> > >> > 1. Unify the Source interface in Flink for batch and stream.
> > >> > 2. Significantly reduce the work for developers to develop new
> source
> > >> > connectors.
> > >> > 3. Provide a common abstraction for all the sources, as well as a
> > >> mechanism
> > >> > to allow source subtasks to coordinate among themselves.
> > >> >
> > >> > The vote will last for at least 72 hours, following the consensus
> > voting
> > >> > process.
> > >> >
> > >> > FLIP wiki:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > >> >
> > >> > Discussion thread:
> > >> >
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jiangjie (Becket) Qin
> > >> >
> > >>
> > >
> >
>

Jingsong Li

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (non-binding), thanks for driving.
FLIP-27 is the basis of a lot of follow-up work.

Best,
Jingsong Lee

On Tue, Feb 4, 2020 at 10:26 AM Jark Wu <[hidden email]> wrote:

> Thanks for driving this Becket!
>
> +1 from my side.
>
> Cheers,
> Jark
>
> On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:
>
> > +1, thanks for the efforts Becket!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]> wrote:
> >
> > > Bump up the thread.
> > >
> > > On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]>
> > wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > I'd like to resume the voting thread for FlIP-27.
> > > >
> > > > Please note that the FLIP wiki has been updated to reflect the latest
> > > > discussions in the discussion thread.
> > > >
> > > > To avoid confusion, I'll only count the votes casted after this
> point.
> > > >
> > > > FLIP wiki:
> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > > > %3A+Refactor+Source+Interface
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting
> > > > process.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > >> +1 (binding), and looking forward to seeing the new interface in the
> > > >> master.
> > > >>
> > > >> Best,
> > > >> Jincheng
> > > >>
> > > >> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > I would like to start the vote for FLIP-27 which proposes to
> > > introduce a
> > > >> > new Source connector interface to address a few problems in the
> > > existing
> > > >> > source connector. The main goals of the the FLIP are following:
> > > >> >
> > > >> > 1. Unify the Source interface in Flink for batch and stream.
> > > >> > 2. Significantly reduce the work for developers to develop new
> > source
> > > >> > connectors.
> > > >> > 3. Provide a common abstraction for all the sources, as well as a
> > > >> mechanism
> > > >> > to allow source subtasks to coordinate among themselves.
> > > >> >
> > > >> > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > >> > process.
> > > >> >
> > > >> > FLIP wiki:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > > >> >
> > > >> > Discussion thread:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jiangjie (Becket) Qin
> > > >> >
> > > >>
> > > >
> > >
> >
>

--
Best, Jingsong Lee

Danny Chan

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (non-binding), thanks for the great work !

Best,
Danny Chan
在 2020年2月4日 +0800 AM11:20，[hidden email]，写道：
>
> +1 (non-binding), thanks for driving.
> FLIP-27 is the basis of a lot of follow-up work.

Wei Zhong-2

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (non-binding), thanks for driving this!

Best,
Wei

> 在 2020年2月4日，11:47，Danny Chan <[hidden email]> 写道：
>
> +1 (non-binding), thanks for the great work !
>
> Best,
> Danny Chan
> 在 2020年2月4日 +0800 AM11:20，[hidden email]，写道：
>>
>> +1 (non-binding), thanks for driving.
>> FLIP-27 is the basis of a lot of follow-up work.

Guowei Ma

Re: [VOTE] FLIP-27 - Refactor Source Interface

In reply to this post by Jingsong Li

+1 (non-binding), thanks for driving.

Best,
Guowei

Jingsong Li <[hidden email]> 于2020年2月4日周二上午11:20写道：

> +1 (non-binding), thanks for driving.
> FLIP-27 is the basis of a lot of follow-up work.
>
> Best,
> Jingsong Lee
>
> On Tue, Feb 4, 2020 at 10:26 AM Jark Wu <[hidden email]> wrote:
>
> > Thanks for driving this Becket!
> >
> > +1 from my side.
> >
> > Cheers,
> > Jark
> >
> > On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:
> >
> > > +1, thanks for the efforts Becket!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]> wrote:
> > >
> > > > Bump up the thread.
> > > >
> > > > On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi Folks,
> > > > >
> > > > > I'd like to resume the voting thread for FlIP-27.
> > > > >
> > > > > Please note that the FLIP wiki has been updated to reflect the
> latest
> > > > > discussions in the discussion thread.
> > > > >
> > > > > To avoid confusion, I'll only count the votes casted after this
> > point.
> > > > >
> > > > > FLIP wiki:
> > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > > > > %3A+Refactor+Source+Interface
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > > > process.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > >> +1 (binding), and looking forward to seeing the new interface in
> the
> > > > >> master.
> > > > >>
> > > > >> Best,
> > > > >> Jincheng
> > > > >>
> > > > >> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> > > > >>
> > > > >> > Hi all,
> > > > >> >
> > > > >> > I would like to start the vote for FLIP-27 which proposes to
> > > > introduce a
> > > > >> > new Source connector interface to address a few problems in the
> > > > existing
> > > > >> > source connector. The main goals of the the FLIP are following:
> > > > >> >
> > > > >> > 1. Unify the Source interface in Flink for batch and stream.
> > > > >> > 2. Significantly reduce the work for developers to develop new
> > > source
> > > > >> > connectors.
> > > > >> > 3. Provide a common abstraction for all the sources, as well as
> a
> > > > >> mechanism
> > > > >> > to allow source subtasks to coordinate among themselves.
> > > > >> >
> > > > >> > The vote will last for at least 72 hours, following the
> consensus
> > > > voting
> > > > >> > process.
> > > > >> >
> > > > >> > FLIP wiki:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > > > >> >
> > > > >> > Discussion thread:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Jiangjie (Becket) Qin
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

Zhijiang(wangzhijiang999)

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (binding), we are waiting too long for it. :)

Best,
Zhijiang

------------------------------------------------------------------
From:Guowei Ma <[hidden email]>
Send Time:2020 Feb. 4 (Tue.) 12:34
To:dev <[hidden email]>
Subject:Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (non-binding), thanks for driving.

Best,
Guowei

Jingsong Li <[hidden email]> 于2020年2月4日周二上午11:20写道：

Piotr Nowojski-3

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (binding)

Piotrek

> On 4 Feb 2020, at 05:39, Zhijiang <[hidden email]> wrote:
>
> +1 (binding), we are waiting too long for it. :)
>
> Best,
> Zhijiang
>
>
> ------------------------------------------------------------------
> From:Guowei Ma <[hidden email]>
> Send Time:2020 Feb. 4 (Tue.) 12:34
> To:dev <[hidden email]>
> Subject:Re: [VOTE] FLIP-27 - Refactor Source Interface
>
> +1 (non-binding), thanks for driving.
>
> Best,
> Guowei
>
>
> Jingsong Li <[hidden email]> 于2020年2月4日周二上午11:20写道：
>
>> +1 (non-binding), thanks for driving.
>> FLIP-27 is the basis of a lot of follow-up work.
>>
>> Best,
>> Jingsong Lee
>>
>> On Tue, Feb 4, 2020 at 10:26 AM Jark Wu <[hidden email]> wrote:
>>
>>> Thanks for driving this Becket!
>>>
>>> +1 from my side.
>>>
>>> Cheers,
>>> Jark
>>>
>>> On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:
>>>
>>>> +1, thanks for the efforts Becket!
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>>
>>>> On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]> wrote:
>>>>
>>>>> Bump up the thread.
>>>>>
>>>>> On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]>
>>>> wrote:
>>>>>
>>>>>> Hi Folks,
>>>>>>
>>>>>> I'd like to resume the voting thread for FlIP-27.
>>>>>>
>>>>>> Please note that the FLIP wiki has been updated to reflect the
>> latest
>>>>>> discussions in the discussion thread.
>>>>>>
>>>>>> To avoid confusion, I'll only count the votes casted after this
>>> point.
>>>>>>
>>>>>> FLIP wiki:
>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
>>>>>> %3A+Refactor+Source+Interface
>>>>>>
>>>>>> Discussion thread:
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
>>>>>>
>>>>>> The vote will last for at least 72 hours, following the consensus
>>>> voting
>>>>>> process.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jiangjie (Becket) Qin
>>>>>>
>>>>>> On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
>>> [hidden email]
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 (binding), and looking forward to seeing the new interface in
>> the
>>>>>>> master.
>>>>>>>
>>>>>>> Best,
>>>>>>> Jincheng
>>>>>>>
>>>>>>> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I would like to start the vote for FLIP-27 which proposes to
>>>>> introduce a
>>>>>>>> new Source connector interface to address a few problems in the
>>>>> existing
>>>>>>>> source connector. The main goals of the the FLIP are following:
>>>>>>>>
>>>>>>>> 1. Unify the Source interface in Flink for batch and stream.
>>>>>>>> 2. Significantly reduce the work for developers to develop new
>>>> source
>>>>>>>> connectors.
>>>>>>>> 3. Provide a common abstraction for all the sources, as well as
>> a
>>>>>>> mechanism
>>>>>>>> to allow source subtasks to coordinate among themselves.
>>>>>>>>
>>>>>>>> The vote will last for at least 72 hours, following the
>> consensus
>>>>> voting
>>>>>>>> process.
>>>>>>>>
>>>>>>>> FLIP wiki:
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
>>>>>>>>
>>>>>>>> Discussion thread:
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Jiangjie (Becket) Qin
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best, Jingsong Lee
>>
>

Becket Qin

Re: [VOTE] FLIP-27 - Refactor Source Interface

Thanks everyone for voting. The voting result is following:

+1 (Binding): 5 (Yu, Jark, Zhijiang, Piotr, Becket)

+1 (Non-binding): 4 (Jingsong, Danny, Wei, Guowei)

-1: 0

FLIP-27 has passed.

Thanks,

Jiangjie (Becket) Qin

On Tue, Feb 4, 2020 at 3:42 PM Piotr Nowojski <[hidden email]> wrote:

> +1 (binding)
>
> Piotrek
>
> > On 4 Feb 2020, at 05:39, Zhijiang <[hidden email]>
> wrote:
> >
> > +1 (binding), we are waiting too long for it. :)
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Guowei Ma <[hidden email]>
> > Send Time:2020 Feb. 4 (Tue.) 12:34
> > To:dev <[hidden email]>
> > Subject:Re: [VOTE] FLIP-27 - Refactor Source Interface
> >
> > +1 (non-binding), thanks for driving.
> >
> > Best,
> > Guowei
> >
> >
> > Jingsong Li <[hidden email]> 于2020年2月4日周二上午11:20写道：
> >
> >> +1 (non-binding), thanks for driving.
> >> FLIP-27 is the basis of a lot of follow-up work.
> >>
> >> Best,
> >> Jingsong Lee
> >>
> >> On Tue, Feb 4, 2020 at 10:26 AM Jark Wu <[hidden email]> wrote:
> >>
> >>> Thanks for driving this Becket!
> >>>
> >>> +1 from my side.
> >>>
> >>> Cheers,
> >>> Jark
> >>>
> >>> On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:
> >>>
> >>>> +1, thanks for the efforts Becket!
> >>>>
> >>>> Best Regards,
> >>>> Yu
> >>>>
> >>>>
> >>>> On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]> wrote:
> >>>>
> >>>>> Bump up the thread.
> >>>>>
> >>>>> On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]>
> >>>> wrote:
> >>>>>
> >>>>>> Hi Folks,
> >>>>>>
> >>>>>> I'd like to resume the voting thread for FlIP-27.
> >>>>>>
> >>>>>> Please note that the FLIP wiki has been updated to reflect the
> >> latest
> >>>>>> discussions in the discussion thread.
> >>>>>>
> >>>>>> To avoid confusion, I'll only count the votes casted after this
> >>> point.
> >>>>>>
> >>>>>> FLIP wiki:
> >>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> >>>>>> %3A+Refactor+Source+Interface
> >>>>>>
> >>>>>> Discussion thread:
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> >>>>>>
> >>>>>> The vote will last for at least 72 hours, following the consensus
> >>>> voting
> >>>>>> process.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jiangjie (Becket) Qin
> >>>>>>
> >>>>>> On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
> >>> [hidden email]
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> +1 (binding), and looking forward to seeing the new interface in
> >> the
> >>>>>>> master.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Jincheng
> >>>>>>>
> >>>>>>> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> >>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> I would like to start the vote for FLIP-27 which proposes to
> >>>>> introduce a
> >>>>>>>> new Source connector interface to address a few problems in the
> >>>>> existing
> >>>>>>>> source connector. The main goals of the the FLIP are following:
> >>>>>>>>
> >>>>>>>> 1. Unify the Source interface in Flink for batch and stream.
> >>>>>>>> 2. Significantly reduce the work for developers to develop new
> >>>> source
> >>>>>>>> connectors.
> >>>>>>>> 3. Provide a common abstraction for all the sources, as well as
> >> a
> >>>>>>> mechanism
> >>>>>>>> to allow source subtasks to coordinate among themselves.
> >>>>>>>>
> >>>>>>>> The vote will last for at least 72 hours, following the
> >> consensus
> >>>>> voting
> >>>>>>>> process.
> >>>>>>>>
> >>>>>>>> FLIP wiki:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >>>>>>>>
> >>>>>>>> Discussion thread:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jiangjie (Becket) Qin
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best, Jingsong Lee
> >>
> >
>
>

Stephan Ewen

Re: [VOTE] FLIP-27 - Refactor Source Interface

+1 (binding) (belated)

Quick addendum to clarify some questions from recent discussions in other
threads:

- The core interfaces (Source, SourceReader, Enumerator) and the core
architecture (Enumerator as coordinators on the JobManager, SourceReaders
in Tasks) seem to have no open questions

- Some open questions remained about whether the proposed base
implementation (SplitReader) fits all possible sources (like block-based
file readers).
- Even if that turns out to not be the case, this FLIP does not prevent
the creation of another base implementation. That would be completely in
line with this FLIP and exactly the reason why the FLIP proposes the
separation between the simpler core interfaces (SourceReader) and the base
implementations, of which the first one (so that we can start to
immediately build some connectors) is the SplitReader.

On Fri, Feb 7, 2020 at 9:47 AM Becket Qin <[hidden email]> wrote:

> Thanks everyone for voting. The voting result is following:
>
> +1 (Binding): 5 (Yu, Jark, Zhijiang, Piotr, Becket)
>
> +1 (Non-binding): 4 (Jingsong, Danny, Wei, Guowei)
>
> -1: 0
>
> FLIP-27 has passed.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Feb 4, 2020 at 3:42 PM Piotr Nowojski <[hidden email]> wrote:
>
> > +1 (binding)
> >
> > Piotrek
> >
> > > On 4 Feb 2020, at 05:39, Zhijiang <[hidden email]>
> > wrote:
> > >
> > > +1 (binding), we are waiting too long for it. :)
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > ------------------------------------------------------------------
> > > From:Guowei Ma <[hidden email]>
> > > Send Time:2020 Feb. 4 (Tue.) 12:34
> > > To:dev <[hidden email]>
> > > Subject:Re: [VOTE] FLIP-27 - Refactor Source Interface
> > >
> > > +1 (non-binding), thanks for driving.
> > >
> > > Best,
> > > Guowei
> > >
> > >
> > > Jingsong Li <[hidden email]> 于2020年2月4日周二上午11:20写道：
> > >
> > >> +1 (non-binding), thanks for driving.
> > >> FLIP-27 is the basis of a lot of follow-up work.
> > >>
> > >> Best,
> > >> Jingsong Lee
> > >>
> > >> On Tue, Feb 4, 2020 at 10:26 AM Jark Wu <[hidden email]> wrote:
> > >>
> > >>> Thanks for driving this Becket!
> > >>>
> > >>> +1 from my side.
> > >>>
> > >>> Cheers,
> > >>> Jark
> > >>>
> > >>> On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:
> > >>>
> > >>>> +1, thanks for the efforts Becket!
> > >>>>
> > >>>> Best Regards,
> > >>>> Yu
> > >>>>
> > >>>>
> > >>>> On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]>
> wrote:
> > >>>>
> > >>>>> Bump up the thread.
> > >>>>>
> > >>>>> On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <[hidden email]>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hi Folks,
> > >>>>>>
> > >>>>>> I'd like to resume the voting thread for FlIP-27.
> > >>>>>>
> > >>>>>> Please note that the FLIP wiki has been updated to reflect the
> > >> latest
> > >>>>>> discussions in the discussion thread.
> > >>>>>>
> > >>>>>> To avoid confusion, I'll only count the votes casted after this
> > >>> point.
> > >>>>>>
> > >>>>>> FLIP wiki:
> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > >>>>>> %3A+Refactor+Source+Interface
> > >>>>>>
> > >>>>>> Discussion thread:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> > >>>>>>
> > >>>>>> The vote will last for at least 72 hours, following the consensus
> > >>>> voting
> > >>>>>> process.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Jiangjie (Becket) Qin
> > >>>>>>
> > >>>>>> On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
> > >>> [hidden email]
> > >>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> +1 (binding), and looking forward to seeing the new interface in
> > >> the
> > >>>>>>> master.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Jincheng
> > >>>>>>>
> > >>>>>>> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> > >>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> I would like to start the vote for FLIP-27 which proposes to
> > >>>>> introduce a
> > >>>>>>>> new Source connector interface to address a few problems in the
> > >>>>> existing
> > >>>>>>>> source connector. The main goals of the the FLIP are following:
> > >>>>>>>>
> > >>>>>>>> 1. Unify the Source interface in Flink for batch and stream.
> > >>>>>>>> 2. Significantly reduce the work for developers to develop new
> > >>>> source
> > >>>>>>>> connectors.
> > >>>>>>>> 3. Provide a common abstraction for all the sources, as well as
> > >> a
> > >>>>>>> mechanism
> > >>>>>>>> to allow source subtasks to coordinate among themselves.
> > >>>>>>>>
> > >>>>>>>> The vote will last for at least 72 hours, following the
> > >> consensus
> > >>>>> voting
> > >>>>>>>> process.
> > >>>>>>>>
> > >>>>>>>> FLIP wiki:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > >>>>>>>>
> > >>>>>>>> Discussion thread:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>>
> > >>>>>>>> Jiangjie (Becket) Qin
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Best, Jingsong Lee
> > >>
> > >
> >
> >
>

Becket Qin

Re: [VOTE] FLIP-27 - Refactor Source Interface

Hi all,

I realized that the FLIP missed the information about *WatermarkOutput*. It
allows users to emit watermark and indicate the idleness of the source, and
is the super class of *SourceOutput.* I have added the related information
to the FLIP wiki. The key information is following:

- *WatermarkOutput** - *An interface for emitting watermark and indicate
idleness of the source.
-

*Watermark* - A new Watermark class will be created in the package
*org.apache.flink.api.common.eventtime. *This class will eventually
replace the existing Watermark in
*org.apache.flink.streaming.api.watermark. *This change allows
flink-core to remain independent of other modules. Given that we will
eventually put all the watermark generation into the Source, this change
will be necessary. Note that this FLIP does not intended to change the
existing way that watermark can be overridden in the DataStream after they
are emitted by the source.

The class / interface definition has also been updated in the FLIP wiki.

Please let us know if you have any concerns regarding this change.

Thanks,

Jiangjie (Becket) Qin

On Fri, Feb 7, 2020 at 9:27 PM Stephan Ewen <[hidden email]> wrote:

> +1 (binding) (belated)
>
> Quick addendum to clarify some questions from recent discussions in other
> threads:
>
> - The core interfaces (Source, SourceReader, Enumerator) and the core
> architecture (Enumerator as coordinators on the JobManager, SourceReaders
> in Tasks) seem to have no open questions
>
> - Some open questions remained about whether the proposed base
> implementation (SplitReader) fits all possible sources (like block-based
> file readers).
> - Even if that turns out to not be the case, this FLIP does not prevent
> the creation of another base implementation. That would be completely in
> line with this FLIP and exactly the reason why the FLIP proposes the
> separation between the simpler core interfaces (SourceReader) and the base
> implementations, of which the first one (so that we can start to
> immediately build some connectors) is the SplitReader.
>
> On Fri, Feb 7, 2020 at 9:47 AM Becket Qin <[hidden email]> wrote:
>
> > Thanks everyone for voting. The voting result is following:
> >
> > +1 (Binding): 5 (Yu, Jark, Zhijiang, Piotr, Becket)
> >
> > +1 (Non-binding): 4 (Jingsong, Danny, Wei, Guowei)
> >
> > -1: 0
> >
> > FLIP-27 has passed.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Tue, Feb 4, 2020 at 3:42 PM Piotr Nowojski <[hidden email]>
> wrote:
> >
> > > +1 (binding)
> > >
> > > Piotrek
> > >
> > > > On 4 Feb 2020, at 05:39, Zhijiang <[hidden email]
> .INVALID>
> > > wrote:
> > > >
> > > > +1 (binding), we are waiting too long for it. :)
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > From:Guowei Ma <[hidden email]>
> > > > Send Time:2020 Feb. 4 (Tue.) 12:34
> > > > To:dev <[hidden email]>
> > > > Subject:Re: [VOTE] FLIP-27 - Refactor Source Interface
> > > >
> > > > +1 (non-binding), thanks for driving.
> > > >
> > > > Best,
> > > > Guowei
> > > >
> > > >
> > > > Jingsong Li <[hidden email]> 于2020年2月4日周二上午11:20写道：
> > > >
> > > >> +1 (non-binding), thanks for driving.
> > > >> FLIP-27 is the basis of a lot of follow-up work.
> > > >>
> > > >> Best,
> > > >> Jingsong Lee
> > > >>
> > > >> On Tue, Feb 4, 2020 at 10:26 AM Jark Wu <[hidden email]> wrote:
> > > >>
> > > >>> Thanks for driving this Becket!
> > > >>>
> > > >>> +1 from my side.
> > > >>>
> > > >>> Cheers,
> > > >>> Jark
> > > >>>
> > > >>> On Mon, 3 Feb 2020 at 18:06, Yu Li <[hidden email]> wrote:
> > > >>>
> > > >>>> +1, thanks for the efforts Becket!
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Yu
> > > >>>>
> > > >>>>
> > > >>>> On Mon, 3 Feb 2020 at 17:52, Becket Qin <[hidden email]>
> > wrote:
> > > >>>>
> > > >>>>> Bump up the thread.
> > > >>>>>
> > > >>>>> On Tue, Jan 21, 2020 at 10:43 AM Becket Qin <
> [hidden email]>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Hi Folks,
> > > >>>>>>
> > > >>>>>> I'd like to resume the voting thread for FlIP-27.
> > > >>>>>>
> > > >>>>>> Please note that the FLIP wiki has been updated to reflect the
> > > >> latest
> > > >>>>>> discussions in the discussion thread.
> > > >>>>>>
> > > >>>>>> To avoid confusion, I'll only count the votes casted after this
> > > >>> point.
> > > >>>>>>
> > > >>>>>> FLIP wiki:
> > > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > > >>>>>> %3A+Refactor+Source+Interface
> > > >>>>>>
> > > >>>>>> Discussion thread:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> > > >>>>>>
> > > >>>>>> The vote will last for at least 72 hours, following the
> consensus
> > > >>>> voting
> > > >>>>>> process.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>>
> > > >>>>>> Jiangjie (Becket) Qin
> > > >>>>>>
> > > >>>>>> On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
> > > >>> [hidden email]
> > > >>>>>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> +1 (binding), and looking forward to seeing the new interface
> in
> > > >> the
> > > >>>>>>> master.
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Jincheng
> > > >>>>>>>
> > > >>>>>>> Becket Qin <[hidden email]> 于2019年12月5日周四上午8:05写道：
> > > >>>>>>>
> > > >>>>>>>> Hi all,
> > > >>>>>>>>
> > > >>>>>>>> I would like to start the vote for FLIP-27 which proposes to
> > > >>>>> introduce a
> > > >>>>>>>> new Source connector interface to address a few problems in
> the
> > > >>>>> existing
> > > >>>>>>>> source connector. The main goals of the the FLIP are
> following:
> > > >>>>>>>>
> > > >>>>>>>> 1. Unify the Source interface in Flink for batch and stream.
> > > >>>>>>>> 2. Significantly reduce the work for developers to develop new
> > > >>>> source
> > > >>>>>>>> connectors.
> > > >>>>>>>> 3. Provide a common abstraction for all the sources, as well
> as
> > > >> a
> > > >>>>>>> mechanism
> > > >>>>>>>> to allow source subtasks to coordinate among themselves.
> > > >>>>>>>>
> > > >>>>>>>> The vote will last for at least 72 hours, following the
> > > >> consensus
> > > >>>>> voting
> > > >>>>>>>> process.
> > > >>>>>>>>
> > > >>>>>>>> FLIP wiki:
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > > >>>>>>>>
> > > >>>>>>>> Discussion thread:
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>>
> > > >>>>>>>> Jiangjie (Becket) Qin
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Best, Jingsong Lee
> > > >>
> > > >
> > >
> > >
> >
>