Hi,
I've put down some words about how we would go about adding support for side inputs to the streaming API. People are increasingly asking for such a feature, which would allow joining a main stream with some slowly changing inputs that can be used for enriching data, among other things. The design document is available here: https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit?usp=sharing Any feedback is very welcome! Cheers, Aljoscha |
Hi Folks,
I added a section about "Implementation Details" to the doc, it is still very rough but I'm afraid this is somewhat tricky and might require some changes to how we handle state. As always, feedback is very welcome! Cheers, Aljoscha On Mon, 2 May 2016 at 11:28 Aljoscha Krettek <[hidden email]> wrote: > Hi, > I've put down some words about how we would go about adding support for > side inputs to the streaming API. People are increasingly asking for such a > feature, which would allow joining a main stream with some slowly changing > inputs that can be used for enriching data, among other things. > > The design document is available here: > https://docs.google.com/document/d/1hIgxi2Zchww_5fWUHLoYiXwSBXjv-M5eOv-MKQYN3m4/edit?usp=sharing > > Any feedback is very welcome! > > Cheers, > Aljoscha > |
Hello,
I find this feature really cool because it would allow people to tackle scenarios requiring a more advanced "join" on multiple streams as already stated above. Here you can find my few thoughts about this feature (a.k.a. my attempt to extend Aljoscha's initial document). I hope you will find it interesting and a good starting point for having a new FLIP soon ;-) Cheers, Ventura |
Hi,
thanks for trying to revive the discussion! I added some comments in the doc. Cheers, Aljoscha On Fri, 28 Oct 2016 at 12:05 venturadelmonte <[hidden email]> wrote: > Hello, > > I find this feature really cool because it would allow people to tackle > scenarios requiring a more advanced "join" on multiple streams as > already stated above. > Here > < > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_db61ky7OQy1hw/edit?usp=sharing > > > you can find my few thoughts about this feature (a.k.a. my > attempt to extend Aljoscha's initial document). > I hope you will find it interesting and a good starting point for > having a new FLIP soon ;-) > > Cheers, > Ventura > > > > -- > View this message in context: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For-Streaming-API-tp11529p14287.html > Sent from the Apache Flink Mailing List archive. mailing list archive at > Nabble.com. > |
Hi, all, we are also trying to find one way to do state initialization for
stateful job, which has some external data stored in external storage. Side Input is a great feature meeting the requirement. I am looking forward to a full proposal including runtime implementations. 2016-11-01 21:52 GMT+08:00 Aljoscha Krettek <[hidden email]>: > Hi, > thanks for trying to revive the discussion! I added some comments in the > doc. > > Cheers, > Aljoscha > > On Fri, 28 Oct 2016 at 12:05 venturadelmonte <[hidden email]> > wrote: > > > Hello, > > > > I find this feature really cool because it would allow people to tackle > > scenarios requiring a more advanced "join" on multiple streams as > > already stated above. > > Here > > < > > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_ > db61ky7OQy1hw/edit?usp=sharing > > > > > you can find my few thoughts about this feature (a.k.a. my > > attempt to extend Aljoscha's initial document). > > I hope you will find it interesting and a good starting point for > > having a new FLIP soon ;-) > > > > Cheers, > > Ventura > > > > > > > > -- > > View this message in context: > > http://apache-flink-mailing-list-archive.1008284.n3. > nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For- > Streaming-API-tp11529p14287.html > > Sent from the Apache Flink Mailing List archive. mailing list archive at > > Nabble.com. > > > |
Hello,
I am also interested in this feature for a paper that I'm writing. I have the "slowly evolving side input" case with a complicated custom "update precondition" that would be expressible by a stateful UDF that makes its decisions from looking at the elements of the main stream. Best, Gábor 2016-11-09 3:19 GMT+01:00 吕文龙 <[hidden email]>: > Hi, all, we are also trying to find one way to do state initialization for > stateful job, which has some external data stored in external storage. Side > Input is a great feature meeting the requirement. I am looking forward to a > full proposal including runtime implementations. > > 2016-11-01 21:52 GMT+08:00 Aljoscha Krettek <[hidden email]>: > >> Hi, >> thanks for trying to revive the discussion! I added some comments in the >> doc. >> >> Cheers, >> Aljoscha >> >> On Fri, 28 Oct 2016 at 12:05 venturadelmonte <[hidden email]> >> wrote: >> >> > Hello, >> > >> > I find this feature really cool because it would allow people to tackle >> > scenarios requiring a more advanced "join" on multiple streams as >> > already stated above. >> > Here >> > < >> > https://docs.google.com/document/d/1hqUmrLY_wPTeS5bqG36Qq9P8LeDjZ_ >> db61ky7OQy1hw/edit?usp=sharing >> > > >> > you can find my few thoughts about this feature (a.k.a. my >> > attempt to extend Aljoscha's initial document). >> > I hope you will find it interesting and a good starting point for >> > having a new FLIP soon ;-) >> > >> > Cheers, >> > Ventura >> > >> > >> > >> > -- >> > View this message in context: >> > http://apache-flink-mailing-list-archive.1008284.n3. >> nabble.com/DISCUSS-Add-Side-Input-Broadcast-Set-For- >> Streaming-API-tp11529p14287.html >> > Sent from the Apache Flink Mailing List archive. mailing list archive at >> > Nabble.com. >> > >> |
Free forum by Nabble | Edit this page |