[DISCUSS] Add Side Input/Broadcast Set For Streaming API

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Add Side Input/Broadcast Set For Streaming API

Aljoscha Krettek-2
Hi,
I've put down some words about how we would go about adding support for
side inputs to the streaming API. People are increasingly asking for such a
feature, which would allow joining a main stream with some slowly changing
inputs that can be used for enriching data, among other things.

The design document is available here:
https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit?usp=sharing

Any feedback is very welcome!

Cheers,
Aljoscha
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

Aljoscha Krettek-2
Hi Folks,
I added a section about "Implementation Details" to the doc, it is still
very rough but I'm afraid this is somewhat tricky and might require some
changes to how we handle state.

As always, feedback is very welcome!

Cheers,
Aljoscha

On Mon, 2 May 2016 at 11:28 Aljoscha Krettek <[hidden email]> wrote:

> Hi,
> I've put down some words about how we would go about adding support for
> side inputs to the streaming API. People are increasingly asking for such a
> feature, which would allow joining a main stream with some slowly changing
> inputs that can be used for enriching data, among other things.
>
> The design document is available here:
> https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit?usp=sharing
>
> Any feedback is very welcome!
>
> Cheers,
> Aljoscha
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

venturadelmonte
Hello,

I find this feature really cool because it would allow people to tackle
scenarios requiring a more advanced "join" on multiple streams as
already stated above.
Here you can find my few thoughts about this feature (a.k.a. my
attempt to extend Aljoscha's initial document).
I hope you will find it interesting and a good starting point for
having a new FLIP soon ;-)

Cheers,
Ventura
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

Aljoscha Krettek-2
Hi,
thanks for trying to revive the discussion! I added some comments in the
doc.

Cheers,
Aljoscha

On Fri, 28 Oct 2016 at 12:05 venturadelmonte <[hidden email]>
wrote:

> Hello,
>
> I find this feature really cool because it would allow people to tackle
> scenarios requiring a more advanced "join" on multiple streams as
> already stated above.
> Here
> <
> https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_db61ky7OQy1hw/edit?usp=sharing
> >
> you can find my few thoughts about this feature (a.k.a. my
> attempt to extend Aljoscha's initial document).
> I hope you will find it interesting and a good starting point for
> having a new FLIP soon ;-)
>
> Cheers,
> Ventura
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-Streaming-API-tp11529p14287.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

wenlong.lwl
Hi, all, we are also trying to find one way to do state initialization for
stateful job, which has some external data stored in external storage. Side
Input is a great feature meeting the requirement. I am looking forward to a
full proposal including runtime implementations.

2016-11-01 21:52 GMT+08:00 Aljoscha Krettek <[hidden email]>:

> Hi,
> thanks for trying to revive the discussion! I added some comments in the
> doc.
>
> Cheers,
> Aljoscha
>
> On Fri, 28 Oct 2016 at 12:05 venturadelmonte <[hidden email]>
> wrote:
>
> > Hello,
> >
> > I find this feature really cool because it would allow people to tackle
> > scenarios requiring a more advanced "join" on multiple streams as
> > already stated above.
> > Here
> > <
> > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_
> db61ky7OQy1hw/edit?usp=sharing
> > >
> > you can find my few thoughts about this feature (a.k.a. my
> > attempt to extend Aljoscha's initial document).
> > I hope you will find it interesting and a good starting point for
> > having a new FLIP soon ;-)
> >
> > Cheers,
> > Ventura
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-
> Streaming-API-tp11529p14287.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

Gábor Gévay
Hello,

I am also interested in this feature for a paper that I'm writing.

I have the "slowly evolving side input" case with a complicated custom
"update precondition" that would be expressible by a stateful UDF that
makes its decisions from looking at the elements of the main stream.

Best,
Gábor




2016-11-09 3:19 GMT+01:00 吕文龙 <[hidden email]>:

> Hi, all, we are also trying to find one way to do state initialization for
> stateful job, which has some external data stored in external storage. Side
> Input is a great feature meeting the requirement. I am looking forward to a
> full proposal including runtime implementations.
>
> 2016-11-01 21:52 GMT+08:00 Aljoscha Krettek <[hidden email]>:
>
>> Hi,
>> thanks for trying to revive the discussion! I added some comments in the
>> doc.
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, 28 Oct 2016 at 12:05 venturadelmonte <[hidden email]>
>> wrote:
>>
>> > Hello,
>> >
>> > I find this feature really cool because it would allow people to tackle
>> > scenarios requiring a more advanced "join" on multiple streams as
>> > already stated above.
>> > Here
>> > <
>> > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_
>> db61ky7OQy1hw/edit?usp=sharing
>> > >
>> > you can find my few thoughts about this feature (a.k.a. my
>> > attempt to extend Aljoscha's initial document).
>> > I hope you will find it interesting and a good starting point for
>> > having a new FLIP soon ;-)
>> >
>> > Cheers,
>> > Ventura
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-flink-mailing-list-archive.1008284.n3.
>> nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-
>> Streaming-API-tp11529p14287.html
>> > Sent from the Apache Flink Mailing List archive. mailing list archive at
>> > Nabble.com.
>> >
>>