Proposal about inner join in Flink

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal about inner join in Flink

xingcanc
Hi everyone,

Recently, I drafted a proposal about inner join in Flink (
http://goo.gl/4AdR7h).

This document reviews some related work on the Table/SQL topic and it
provides a relatively complete view about the inner join semantics and
implementation. Besides, I also share my (objective) thoughts about
unifying the batch/stream query processing.

I know there are lots of developers who are interested in this subject.
Please share your ideas and all suggestions are welcome.

Thanks,
Xingcan
Reply | Threaded
Open this post in threaded view
|

Re: Proposal about inner join in Flink

Shaoxuan Wang
Hello Xingcan,
Thanks for the proposal. It seems (I may miss something) the proposed
semantics&approach for unbounded inner join is similar as the one proposed
in FLINK-5878.
I did not create the PR for FLINK-5878, as the implementation for inner
join is closely associated with "Retract" (proposed in FLINK-6047).
I have not completely read through your doc, but the window-joins that you
mentioned are definitely the topics that we are also interested. Will read
it carefully and left comments on your doc. Thanks!

Regards,
Shaoxuan


On Wed, May 17, 2017 at 8:56 PM, Xingcan Cui <[hidden email]> wrote:

> Hi everyone,
>
> Recently, I drafted a proposal about inner join in Flink (
> http://goo.gl/4AdR7h).
>
> This document reviews some related work on the Table/SQL topic and it
> provides a relatively complete view about the inner join semantics and
> implementation. Besides, I also share my (objective) thoughts about
> unifying the batch/stream query processing.
>
> I know there are lots of developers who are interested in this subject.
> Please share your ideas and all suggestions are welcome.
>
> Thanks,
> Xingcan
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal about inner join in Flink

Hongyuhong
Hi Xingcan,
Thanks for the proposal.
I have glanced at the design document but not detailedly. The semantics of Record-to-window Join is already in progress in FLINK-5725,.
It would be great if you can share your ideas about the implementions.

Thanks very much.
Yuhong

-----邮件原件-----
发件人: Shaoxuan Wang [mailto:[hidden email]]
发送时间: 2017年5月17日 22:48
收件人: Dev
主题: Re: Proposal about inner join in Flink

Hello Xingcan,
Thanks for the proposal. It seems (I may miss something) the proposed semantics&approach for unbounded inner join is similar as the one proposed in FLINK-5878.
I did not create the PR for FLINK-5878, as the implementation for inner join is closely associated with "Retract" (proposed in FLINK-6047).
I have not completely read through your doc, but the window-joins that you mentioned are definitely the topics that we are also interested. Will read it carefully and left comments on your doc. Thanks!

Regards,
Shaoxuan


On Wed, May 17, 2017 at 8:56 PM, Xingcan Cui <[hidden email]> wrote:

> Hi everyone,
>
> Recently, I drafted a proposal about inner join in Flink (
> http://goo.gl/4AdR7h).
>
> This document reviews some related work on the Table/SQL topic and it
> provides a relatively complete view about the inner join semantics and
> implementation. Besides, I also share my (objective) thoughts about
> unifying the batch/stream query processing.
>
> I know there are lots of developers who are interested in this subject.
> Please share your ideas and all suggestions are welcome.
>
> Thanks,
> Xingcan
>
Reply | Threaded
Open this post in threaded view
|

Re: Proposal about inner join in Flink

xingcanc
Hi Shaoxuan and Yuhong,

thanks for your feedbacks.

@Shaoxuan, actually the unbounded inner join is just inspired by your
document and the discussions. I've spent a lot of time arranging the
references, but still put the link at a wrong please. Sorry for that and
had it done now. Go back to work... I think there's no need to retract
something for unbounded inner joins (well, maybe we need it when taking the
join as a sub-query). What's your opinion? As for the window-joins, I think
the most difficult part is how to consolidate the semantics with batch
environment. I present a naive solution (by system UDF) for that and not
sure if it will work.

@Yuhong, I've reviewed the PR for FLINK-5725. For the SQL translation part,
I think it's clear and suitable. For the execution part I suppose it can
work and generate precise results. The only point I concerned is the
parallelism. Will it only be executed in a single thread?

Thanks,
Xingcan

On Thu, May 18, 2017 at 11:40 AM, Hongyuhong <[hidden email]> wrote:

> Hi Xingcan,
> Thanks for the proposal.
> I have glanced at the design document but not detailedly. The semantics of
> Record-to-window Join is already in progress in FLINK-5725,.
> It would be great if you can share your ideas about the implementions.
>
> Thanks very much.
> Yuhong
>
> -----邮件原件-----
> 发件人: Shaoxuan Wang [mailto:[hidden email]]
> 发送时间: 2017年5月17日 22:48
> 收件人: Dev
> 主题: Re: Proposal about inner join in Flink
>
> Hello Xingcan,
> Thanks for the proposal. It seems (I may miss something) the proposed
> semantics&approach for unbounded inner join is similar as the one proposed
> in FLINK-5878.
> I did not create the PR for FLINK-5878, as the implementation for inner
> join is closely associated with "Retract" (proposed in FLINK-6047).
> I have not completely read through your doc, but the window-joins that you
> mentioned are definitely the topics that we are also interested. Will read
> it carefully and left comments on your doc. Thanks!
>
> Regards,
> Shaoxuan
>
>
> On Wed, May 17, 2017 at 8:56 PM, Xingcan Cui <[hidden email]> wrote:
>
> > Hi everyone,
> >
> > Recently, I drafted a proposal about inner join in Flink (
> > http://goo.gl/4AdR7h).
> >
> > This document reviews some related work on the Table/SQL topic and it
> > provides a relatively complete view about the inner join semantics and
> > implementation. Besides, I also share my (objective) thoughts about
> > unifying the batch/stream query processing.
> >
> > I know there are lots of developers who are interested in this subject.
> > Please share your ideas and all suggestions are welcome.
> >
> > Thanks,
> > Xingcan
> >
>