Hey guys,
I am trying contain my excitement, but I am failing :) So today I implemented a basic prototype for connecting batch processing and streaming jobs as a good starting point for a 'lambda architecture api'. And most surprisingly it works :) It is very simple so far (currently only works for map operator and chaining is turned off) and some hacking along the way, but I hope you will like it. You can see the commit here: https://github.com/mbalassi/incubator-flink/commit/7bfc16a7803e1f2cc4b38c746af069aab4450637 And a sample job: https://github.com/mbalassi/incubator-flink/blob/7bfc16a7803e1f2cc4b38c746af069aab4450637/flink-addons/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/LambdaTest.java And for the others, it looks something like this: ExecutionEnvironment batchEnv = ExecutionEnvironment. *createLocalEnvironment*(1); StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment. *createLocalEnvironment*(1); DataSet<Long> dataSet = batchEnv.generateSequence(1, 10).map(*new* Increment()); DataStream<Long> dataStream = streamEnv.generateSequence(10, 20); *dataStream.lambdaJoin(dataSet).map(new Increment()).print();* dataSet.print(); * LambdaEnvironment lambdaEnv = new LambdaEnvironment(batchEnv, streamEnv);* * lambdaEnv.execute(":)");* I hope I could get you excited too :) Cheers, Gyula |
Wow, that is in fact quite impressive. :)
On Fri, Sep 26, 2014 at 11:21 PM, Gyula Fóra <[hidden email]> wrote: > Hey guys, > > I am trying contain my excitement, but I am failing :) > > So today I implemented a basic prototype for connecting batch processing > and streaming jobs as a good starting point for a 'lambda architecture > api'. And most surprisingly it works :) It is very simple so far (currently > only works for map operator and chaining is turned off) and some hacking > along the way, but I hope you will like it. > > You can see the commit here: > > https://github.com/mbalassi/incubator-flink/commit/7bfc16a7803e1f2cc4b38c746af069aab4450637 > > And a sample job: > > https://github.com/mbalassi/incubator-flink/blob/7bfc16a7803e1f2cc4b38c746af069aab4450637/flink-addons/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/LambdaTest.java > > And for the others, it looks something like this: > > ExecutionEnvironment batchEnv = ExecutionEnvironment. > *createLocalEnvironment*(1); > StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment. > *createLocalEnvironment*(1); > > DataSet<Long> dataSet = batchEnv.generateSequence(1, 10).map(*new* > Increment()); > > DataStream<Long> dataStream = streamEnv.generateSequence(10, 20); > > *dataStream.lambdaJoin(dataSet).map(new Increment()).print();* > dataSet.print(); > > * LambdaEnvironment lambdaEnv = new LambdaEnvironment(batchEnv, > streamEnv);* > > * lambdaEnv.execute(":)");* > > > I hope I could get you excited too :) > > Cheers, > > Gyula > |
On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]> wrote: >> I hope I could get you excited too :) +1 :-) |
Wow, amazing :-) And so quickly done!
Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <[hidden email]>: > > On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]> wrote: > > >> I hope I could get you excited too :) > > +1 :-) |
Wow so cool!
On Sat, Sep 27, 2014 at 2:52 PM, Stephan Ewen <[hidden email]> wrote: > Wow, amazing :-) And so quickly done! > Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <[hidden email]>: > > > > > On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]> wrote: > > > > >> I hope I could get you excited too :) > > > > +1 :-) > |
+1
Looks nice! On 27 Sep 2014, at 21:04, Kostas Tzoumas <[hidden email]<mailto:[hidden email]>> wrote: Wow so cool! On Sat, Sep 27, 2014 at 2:52 PM, Stephan Ewen <[hidden email]<mailto:[hidden email]>> wrote: Wow, amazing :-) And so quickly done! Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <[hidden email]<mailto:[hidden email]>>: On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]<mailto:[hidden email]>> wrote: I hope I could get you excited too :) +1 :-) |
Free forum by Nabble | Edit this page |