Samoa on Flink

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Samoa on Flink

Vasiliki Kalavri
Hello everyone,

Last week at Strata, I attended a talk about a streaming machine learning
framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
project is going through apache incubation application and currently runs
on Storm, S4 and Samza.

After the talk, I spoke with the guy that presented it and we both thought
that it'd be cool to port Samoa on Flink.

I'm writing about this here because I want to know your thoughts on this
and also because I'm not very familiar with the Streaming-API, so I will
need your input :)
I have already discussed with Paris and there is interest to pick this up
from some students at KTH as well.
As far as I understand, Samoa expects an API similar to that of Storm and
from there the integration should be pretty straight-forward.

So, do you think it would be feasible and -most importantly- worth it?

Cheers,
Vasia.
Reply | Threaded
Open this post in threaded view
|

Re: Samoa on Flink

Paris Carbone
Hey Vasia,

I really like the idea and it seems we have some volunteers from our side already!
It is in my opinion a starter task since it requires to create a simple adapter for flink as well as mapping abstractions to our streaming topology API in a similar fashion (in its simplest form just using several flatmaps).

It will serve as a suitable common layer to make some comparisons between flink-streaming to storm, s4 and samoa on incremental machine learning tasks.

Paris

> On 27 Nov 2014, at 13:59, Vasiliki Kalavri <[hidden email]> wrote:
>
> Hello everyone,
>
> Last week at Strata, I attended a talk about a streaming machine learning
> framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> project is going through apache incubation application and currently runs
> on Storm, S4 and Samza.
>
> After the talk, I spoke with the guy that presented it and we both thought
> that it'd be cool to port Samoa on Flink.
>
> I'm writing about this here because I want to know your thoughts on this
> and also because I'm not very familiar with the Streaming-API, so I will
> need your input :)
> I have already discussed with Paris and there is interest to pick this up
> from some students at KTH as well.
> As far as I understand, Samoa expects an API similar to that of Storm and
> from there the integration should be pretty straight-forward.
>
> So, do you think it would be feasible and -most importantly- worth it?
>
> Cheers,
> Vasia.

Reply | Threaded
Open this post in threaded view
|

Re: Samoa on Flink

Márton Balassi
Hey,

Thanks Vasia, great idea.
When we've started the streaming development the first sketch API we had is
was actually really similar to Storm's API a couple of versions back. I've
talked to Gyula about the issue when we've first heard about SAMOA ago and
we are confident that porting Samoa to Flink streaming shoul be definitely
managable and really useful as Paris mentions.

Luckily they have ASF2 license so the legal cooperation is clear. The rest
is on us and fortunately Paris might have a candidate already to kick off
the development. :)

Cheers,

Marton

On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <[hidden email]> wrote:

> Hey Vasia,
>
> I really like the idea and it seems we have some volunteers from our side
> already!
> It is in my opinion a starter task since it requires to create a simple
> adapter for flink as well as mapping abstractions to our streaming topology
> API in a similar fashion (in its simplest form just using several flatmaps).
>
> It will serve as a suitable common layer to make some comparisons between
> flink-streaming to storm, s4 and samoa on incremental machine learning
> tasks.
>
> Paris
>
> > On 27 Nov 2014, at 13:59, Vasiliki Kalavri <[hidden email]>
> wrote:
> >
> > Hello everyone,
> >
> > Last week at Strata, I attended a talk about a streaming machine learning
> > framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> > project is going through apache incubation application and currently runs
> > on Storm, S4 and Samza.
> >
> > After the talk, I spoke with the guy that presented it and we both
> thought
> > that it'd be cool to port Samoa on Flink.
> >
> > I'm writing about this here because I want to know your thoughts on this
> > and also because I'm not very familiar with the Streaming-API, so I will
> > need your input :)
> > I have already discussed with Paris and there is interest to pick this up
> > from some students at KTH as well.
> > As far as I understand, Samoa expects an API similar to that of Storm and
> > from there the integration should be pretty straight-forward.
> >
> > So, do you think it would be feasible and -most importantly- worth it?
> >
> > Cheers,
> > Vasia.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Samoa on Flink

Fabian Hueske
FYI, Samoa was just proposed for Apache Incubation :-)

2014-11-27 15:19 GMT+01:00 Márton Balassi <[hidden email]>:

> Hey,
>
> Thanks Vasia, great idea.
> When we've started the streaming development the first sketch API we had is
> was actually really similar to Storm's API a couple of versions back. I've
> talked to Gyula about the issue when we've first heard about SAMOA ago and
> we are confident that porting Samoa to Flink streaming shoul be definitely
> managable and really useful as Paris mentions.
>
> Luckily they have ASF2 license so the legal cooperation is clear. The rest
> is on us and fortunately Paris might have a candidate already to kick off
> the development. :)
>
> Cheers,
>
> Marton
>
> On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <[hidden email]> wrote:
>
> > Hey Vasia,
> >
> > I really like the idea and it seems we have some volunteers from our side
> > already!
> > It is in my opinion a starter task since it requires to create a simple
> > adapter for flink as well as mapping abstractions to our streaming
> topology
> > API in a similar fashion (in its simplest form just using several
> flatmaps).
> >
> > It will serve as a suitable common layer to make some comparisons between
> > flink-streaming to storm, s4 and samoa on incremental machine learning
> > tasks.
> >
> > Paris
> >
> > > On 27 Nov 2014, at 13:59, Vasiliki Kalavri <[hidden email]>
> > wrote:
> > >
> > > Hello everyone,
> > >
> > > Last week at Strata, I attended a talk about a streaming machine
> learning
> > > framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> > > project is going through apache incubation application and currently
> runs
> > > on Storm, S4 and Samza.
> > >
> > > After the talk, I spoke with the guy that presented it and we both
> > thought
> > > that it'd be cool to port Samoa on Flink.
> > >
> > > I'm writing about this here because I want to know your thoughts on
> this
> > > and also because I'm not very familiar with the Streaming-API, so I
> will
> > > need your input :)
> > > I have already discussed with Paris and there is interest to pick this
> up
> > > from some students at KTH as well.
> > > As far as I understand, Samoa expects an API similar to that of Storm
> and
> > > from there the integration should be pretty straight-forward.
> > >
> > > So, do you think it would be feasible and -most importantly- worth it?
> > >
> > > Cheers,
> > > Vasia.
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Samoa on Flink

Paris Carbone
Great news! Good to be aligned :)

Paris

> On 03 Dec 2014, at 10:24, Fabian Hueske <[hidden email]> wrote:
>
> FYI, Samoa was just proposed for Apache Incubation :-)
>
> 2014-11-27 15:19 GMT+01:00 Márton Balassi <[hidden email]>:
>
>> Hey,
>>
>> Thanks Vasia, great idea.
>> When we've started the streaming development the first sketch API we had is
>> was actually really similar to Storm's API a couple of versions back. I've
>> talked to Gyula about the issue when we've first heard about SAMOA ago and
>> we are confident that porting Samoa to Flink streaming shoul be definitely
>> managable and really useful as Paris mentions.
>>
>> Luckily they have ASF2 license so the legal cooperation is clear. The rest
>> is on us and fortunately Paris might have a candidate already to kick off
>> the development. :)
>>
>> Cheers,
>>
>> Marton
>>
>> On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <[hidden email]> wrote:
>>
>>> Hey Vasia,
>>>
>>> I really like the idea and it seems we have some volunteers from our side
>>> already!
>>> It is in my opinion a starter task since it requires to create a simple
>>> adapter for flink as well as mapping abstractions to our streaming
>> topology
>>> API in a similar fashion (in its simplest form just using several
>> flatmaps).
>>>
>>> It will serve as a suitable common layer to make some comparisons between
>>> flink-streaming to storm, s4 and samoa on incremental machine learning
>>> tasks.
>>>
>>> Paris
>>>
>>>> On 27 Nov 2014, at 13:59, Vasiliki Kalavri <[hidden email]>
>>> wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> Last week at Strata, I attended a talk about a streaming machine
>> learning
>>>> framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
>>>> project is going through apache incubation application and currently
>> runs
>>>> on Storm, S4 and Samza.
>>>>
>>>> After the talk, I spoke with the guy that presented it and we both
>>> thought
>>>> that it'd be cool to port Samoa on Flink.
>>>>
>>>> I'm writing about this here because I want to know your thoughts on
>> this
>>>> and also because I'm not very familiar with the Streaming-API, so I
>> will
>>>> need your input :)
>>>> I have already discussed with Paris and there is interest to pick this
>> up
>>>> from some students at KTH as well.
>>>> As far as I understand, Samoa expects an API similar to that of Storm
>> and
>>>> from there the integration should be pretty straight-forward.
>>>>
>>>> So, do you think it would be feasible and -most importantly- worth it?
>>>>
>>>> Cheers,
>>>> Vasia.
>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Samoa on Flink

Vasiliki Kalavri
FYI, we'll be working on this here:
https://github.com/senorcarbone/samoa/tree/flink
Feel free to provide input :))

Cheers,
V.

On 3 December 2014 at 11:56, Paris Carbone <[hidden email]> wrote:

> Great news! Good to be aligned :)
>
> Paris
>
> > On 03 Dec 2014, at 10:24, Fabian Hueske <[hidden email]> wrote:
> >
> > FYI, Samoa was just proposed for Apache Incubation :-)
> >
> > 2014-11-27 15:19 GMT+01:00 Márton Balassi <[hidden email]>:
> >
> >> Hey,
> >>
> >> Thanks Vasia, great idea.
> >> When we've started the streaming development the first sketch API we
> had is
> >> was actually really similar to Storm's API a couple of versions back.
> I've
> >> talked to Gyula about the issue when we've first heard about SAMOA ago
> and
> >> we are confident that porting Samoa to Flink streaming shoul be
> definitely
> >> managable and really useful as Paris mentions.
> >>
> >> Luckily they have ASF2 license so the legal cooperation is clear. The
> rest
> >> is on us and fortunately Paris might have a candidate already to kick
> off
> >> the development. :)
> >>
> >> Cheers,
> >>
> >> Marton
> >>
> >> On Thu, Nov 27, 2014 at 2:05 PM, Paris Carbone <[hidden email]> wrote:
> >>
> >>> Hey Vasia,
> >>>
> >>> I really like the idea and it seems we have some volunteers from our
> side
> >>> already!
> >>> It is in my opinion a starter task since it requires to create a simple
> >>> adapter for flink as well as mapping abstractions to our streaming
> >> topology
> >>> API in a similar fashion (in its simplest form just using several
> >> flatmaps).
> >>>
> >>> It will serve as a suitable common layer to make some comparisons
> between
> >>> flink-streaming to storm, s4 and samoa on incremental machine learning
> >>> tasks.
> >>>
> >>> Paris
> >>>
> >>>> On 27 Nov 2014, at 13:59, Vasiliki Kalavri <[hidden email]
> >
> >>> wrote:
> >>>>
> >>>> Hello everyone,
> >>>>
> >>>> Last week at Strata, I attended a talk about a streaming machine
> >> learning
> >>>> framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The
> >>>> project is going through apache incubation application and currently
> >> runs
> >>>> on Storm, S4 and Samza.
> >>>>
> >>>> After the talk, I spoke with the guy that presented it and we both
> >>> thought
> >>>> that it'd be cool to port Samoa on Flink.
> >>>>
> >>>> I'm writing about this here because I want to know your thoughts on
> >> this
> >>>> and also because I'm not very familiar with the Streaming-API, so I
> >> will
> >>>> need your input :)
> >>>> I have already discussed with Paris and there is interest to pick this
> >> up
> >>>> from some students at KTH as well.
> >>>> As far as I understand, Samoa expects an API similar to that of Storm
> >> and
> >>>> from there the integration should be pretty straight-forward.
> >>>>
> >>>> So, do you think it would be feasible and -most importantly- worth it?
> >>>>
> >>>> Cheers,
> >>>> Vasia.
> >>>
> >>>
> >>
>
>