Storm compatibility layer for Flink (first beta available)

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Storm compatibility layer for Flink (first beta available)

Matthias J. Sax
Hi @all,

I started to work on an compatibility layer to run Storm Topologies on
Flink. I just pushed a first beta:
https://github.com/mjsax/flink/tree/flink-storm-compatibility

Please check it out, and let me know how you like it. In this first
version, I tried to code without changing too many things in the
existing code. However, a deeper integration would be nice.

There are many things that do not work yet. Currently, only shuffle and
fieldsGrouping is supported (and only Storm's default output stream).
Furthermore, topologies must be "simple", ie, they cannot be configured
with a Config object and Storm meta information (ie, TopologyContext) is
also not there.

The layer can be used to integrate existing Spouts and/or Bolts as Flink
operators into a regular Flink program. Furthermore, a whole Storm
topology can be executed by switching from TopologyBuilder to
FlinkTopologyBuilder and from LocalCluster/NimbusClient/StormSubmitter
to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.

Examples of both cases are given as ITCases.


Cheers,
  Matthias



signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Robert Metzger
Hey Matthias,

a Storm compatibility layer sounds really great!

I'll soon take a closer look into the code, but the features you're listing
sound really amazing! Since the code has already testcases included, I'm
open to merging a first stable version and then continue the development of
the feature from Flink's git repository.




On Thu, Apr 2, 2015 at 7:31 PM, Matthias J. Sax <
[hidden email]> wrote:

> Hi @all,
>
> I started to work on an compatibility layer to run Storm Topologies on
> Flink. I just pushed a first beta:
> https://github.com/mjsax/flink/tree/flink-storm-compatibility
>
> Please check it out, and let me know how you like it. In this first
> version, I tried to code without changing too many things in the
> existing code. However, a deeper integration would be nice.
>
> There are many things that do not work yet. Currently, only shuffle and
> fieldsGrouping is supported (and only Storm's default output stream).
> Furthermore, topologies must be "simple", ie, they cannot be configured
> with a Config object and Storm meta information (ie, TopologyContext) is
> also not there.
>
> The layer can be used to integrate existing Spouts and/or Bolts as Flink
> operators into a regular Flink program. Furthermore, a whole Storm
> topology can be executed by switching from TopologyBuilder to
> FlinkTopologyBuilder and from LocalCluster/NimbusClient/StormSubmitter
> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
>
> Examples of both cases are given as ITCases.
>
>
> Cheers,
>   Matthias
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Henry Saputra
In reply to this post by Matthias J. Sax
HI Matthias,

Where do you put the code for the Storm compatibility? Under streams
module directory?

- Henry

On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
<[hidden email]> wrote:

> Hi @all,
>
> I started to work on an compatibility layer to run Storm Topologies on
> Flink. I just pushed a first beta:
> https://github.com/mjsax/flink/tree/flink-storm-compatibility
>
> Please check it out, and let me know how you like it. In this first
> version, I tried to code without changing too many things in the
> existing code. However, a deeper integration would be nice.
>
> There are many things that do not work yet. Currently, only shuffle and
> fieldsGrouping is supported (and only Storm's default output stream).
> Furthermore, topologies must be "simple", ie, they cannot be configured
> with a Config object and Storm meta information (ie, TopologyContext) is
> also not there.
>
> The layer can be used to integrate existing Spouts and/or Bolts as Flink
> operators into a regular Flink program. Furthermore, a whole Storm
> topology can be executed by switching from TopologyBuilder to
> FlinkTopologyBuilder and from LocalCluster/NimbusClient/StormSubmitter
> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
>
> Examples of both cases are given as ITCases.
>
>
> Cheers,
>   Matthias
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Robert Metzger
Hey Henry,

you can check out the files here:
https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
... so yes, they are located in the flink-streaming directory .. which is a
good place for now.
Once we move flink-streaming out of staging, we might want to keep the
storm compat in staging.


On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <[hidden email]>
wrote:

> HI Matthias,
>
> Where do you put the code for the Storm compatibility? Under streams
> module directory?
>
> - Henry
>
> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> <[hidden email]> wrote:
> > Hi @all,
> >
> > I started to work on an compatibility layer to run Storm Topologies on
> > Flink. I just pushed a first beta:
> > https://github.com/mjsax/flink/tree/flink-storm-compatibility
> >
> > Please check it out, and let me know how you like it. In this first
> > version, I tried to code without changing too many things in the
> > existing code. However, a deeper integration would be nice.
> >
> > There are many things that do not work yet. Currently, only shuffle and
> > fieldsGrouping is supported (and only Storm's default output stream).
> > Furthermore, topologies must be "simple", ie, they cannot be configured
> > with a Config object and Storm meta information (ie, TopologyContext) is
> > also not there.
> >
> > The layer can be used to integrate existing Spouts and/or Bolts as Flink
> > operators into a regular Flink program. Furthermore, a whole Storm
> > topology can be executed by switching from TopologyBuilder to
> > FlinkTopologyBuilder and from LocalCluster/NimbusClient/StormSubmitter
> > to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> >
> > Examples of both cases are given as ITCases.
> >
> >
> > Cheers,
> >   Matthias
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Fabian Hueske-2
Hi Matthias,

this is really cool!I especially like that you can use Storm code within a
Flink streaming program :-)

One thing that might be good to do rather soon is to collect all your
commits and put them on top of a fresh forked Flink master branch.
When merging we cannot change the history and try to put only fast-forward
commits on top of the master branch.
As time goes on it becomes more likely that you run into merge issues when
cherry-picking the commits.

2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]>:

> Hey Henry,
>
> you can check out the files here:
>
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> ... so yes, they are located in the flink-streaming directory .. which is a
> good place for now.
> Once we move flink-streaming out of staging, we might want to keep the
> storm compat in staging.
>
>
> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <[hidden email]>
> wrote:
>
> > HI Matthias,
> >
> > Where do you put the code for the Storm compatibility? Under streams
> > module directory?
> >
> > - Henry
> >
> > On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> > <[hidden email]> wrote:
> > > Hi @all,
> > >
> > > I started to work on an compatibility layer to run Storm Topologies on
> > > Flink. I just pushed a first beta:
> > > https://github.com/mjsax/flink/tree/flink-storm-compatibility
> > >
> > > Please check it out, and let me know how you like it. In this first
> > > version, I tried to code without changing too many things in the
> > > existing code. However, a deeper integration would be nice.
> > >
> > > There are many things that do not work yet. Currently, only shuffle and
> > > fieldsGrouping is supported (and only Storm's default output stream).
> > > Furthermore, topologies must be "simple", ie, they cannot be configured
> > > with a Config object and Storm meta information (ie, TopologyContext)
> is
> > > also not there.
> > >
> > > The layer can be used to integrate existing Spouts and/or Bolts as
> Flink
> > > operators into a regular Flink program. Furthermore, a whole Storm
> > > topology can be executed by switching from TopologyBuilder to
> > > FlinkTopologyBuilder and from LocalCluster/NimbusClient/StormSubmitter
> > > to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> > >
> > > Examples of both cases are given as ITCases.
> > >
> > >
> > > Cheers,
> > >   Matthias
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Márton Balassi
Hey Mathias,

Thanks, this is a really nice contribution. I just scrolled through the
code, but I really like it and big thanks for the the tests for the
examples.

The rebase Fabian suggested would help a lot when merging.



On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]> wrote:

> Hi Matthias,
>
> this is really cool!I especially like that you can use Storm code within a
> Flink streaming program :-)
>
> One thing that might be good to do rather soon is to collect all your
> commits and put them on top of a fresh forked Flink master branch.
> When merging we cannot change the history and try to put only fast-forward
> commits on top of the master branch.
> As time goes on it becomes more likely that you run into merge issues when
> cherry-picking the commits.
>
> 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]>:
>
> > Hey Henry,
> >
> > you can check out the files here:
> >
> >
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> > ... so yes, they are located in the flink-streaming directory .. which
> is a
> > good place for now.
> > Once we move flink-streaming out of staging, we might want to keep the
> > storm compat in staging.
> >
> >
> > On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <[hidden email]>
> > wrote:
> >
> > > HI Matthias,
> > >
> > > Where do you put the code for the Storm compatibility? Under streams
> > > module directory?
> > >
> > > - Henry
> > >
> > > On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> > > <[hidden email]> wrote:
> > > > Hi @all,
> > > >
> > > > I started to work on an compatibility layer to run Storm Topologies
> on
> > > > Flink. I just pushed a first beta:
> > > > https://github.com/mjsax/flink/tree/flink-storm-compatibility
> > > >
> > > > Please check it out, and let me know how you like it. In this first
> > > > version, I tried to code without changing too many things in the
> > > > existing code. However, a deeper integration would be nice.
> > > >
> > > > There are many things that do not work yet. Currently, only shuffle
> and
> > > > fieldsGrouping is supported (and only Storm's default output stream).
> > > > Furthermore, topologies must be "simple", ie, they cannot be
> configured
> > > > with a Config object and Storm meta information (ie, TopologyContext)
> > is
> > > > also not there.
> > > >
> > > > The layer can be used to integrate existing Spouts and/or Bolts as
> > Flink
> > > > operators into a regular Flink program. Furthermore, a whole Storm
> > > > topology can be executed by switching from TopologyBuilder to
> > > > FlinkTopologyBuilder and from
> LocalCluster/NimbusClient/StormSubmitter
> > > > to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> > > >
> > > > Examples of both cases are given as ITCases.
> > > >
> > > >
> > > > Cheers,
> > > >   Matthias
> > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Gyula Fóra-2
This sounds amazing :) thanks Matthias!

Tomorrow I will spend some time to look through your work and give some
comments.

Also I would love to help with this effort so once we merge an initial
prototype let's open some Jiras and I will pick some up :)

Gyula

On Thursday, April 2, 2015, Márton Balassi <[hidden email]> wrote:

> Hey Mathias,
>
> Thanks, this is a really nice contribution. I just scrolled through the
> code, but I really like it and big thanks for the the tests for the
> examples.
>
> The rebase Fabian suggested would help a lot when merging.
>
>
>
> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]
> <javascript:;>> wrote:
>
> > Hi Matthias,
> >
> > this is really cool!I especially like that you can use Storm code within
> a
> > Flink streaming program :-)
> >
> > One thing that might be good to do rather soon is to collect all your
> > commits and put them on top of a fresh forked Flink master branch.
> > When merging we cannot change the history and try to put only
> fast-forward
> > commits on top of the master branch.
> > As time goes on it becomes more likely that you run into merge issues
> when
> > cherry-picking the commits.
> >
> > 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]
> <javascript:;>>:
> >
> > > Hey Henry,
> > >
> > > you can check out the files here:
> > >
> > >
> >
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> > > ... so yes, they are located in the flink-streaming directory .. which
> > is a
> > > good place for now.
> > > Once we move flink-streaming out of staging, we might want to keep the
> > > storm compat in staging.
> > >
> > >
> > > On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <[hidden email]
> <javascript:;>>
> > > wrote:
> > >
> > > > HI Matthias,
> > > >
> > > > Where do you put the code for the Storm compatibility? Under streams
> > > > module directory?
> > > >
> > > > - Henry
> > > >
> > > > On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> > > > <[hidden email] <javascript:;>> wrote:
> > > > > Hi @all,
> > > > >
> > > > > I started to work on an compatibility layer to run Storm Topologies
> > on
> > > > > Flink. I just pushed a first beta:
> > > > > https://github.com/mjsax/flink/tree/flink-storm-compatibility
> > > > >
> > > > > Please check it out, and let me know how you like it. In this first
> > > > > version, I tried to code without changing too many things in the
> > > > > existing code. However, a deeper integration would be nice.
> > > > >
> > > > > There are many things that do not work yet. Currently, only shuffle
> > and
> > > > > fieldsGrouping is supported (and only Storm's default output
> stream).
> > > > > Furthermore, topologies must be "simple", ie, they cannot be
> > configured
> > > > > with a Config object and Storm meta information (ie,
> TopologyContext)
> > > is
> > > > > also not there.
> > > > >
> > > > > The layer can be used to integrate existing Spouts and/or Bolts as
> > > Flink
> > > > > operators into a regular Flink program. Furthermore, a whole Storm
> > > > > topology can be executed by switching from TopologyBuilder to
> > > > > FlinkTopologyBuilder and from
> > LocalCluster/NimbusClient/StormSubmitter
> > > > > to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> > > > >
> > > > > Examples of both cases are given as ITCases.
> > > > >
> > > > >
> > > > > Cheers,
> > > > >   Matthias
> > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Paris Carbone
That’s pretty nice Matthias, we could use a compositional API in streaming that many people are familiar with.
I can also help in some parts, I see some issues we already encountered while creating the samoa adapter (eg. dealing with circles in the topology). Thanks again for initiating this!

Paris

> On 02 Apr 2015, at 23:14, Gyula Fóra <[hidden email]> wrote:
>
> This sounds amazing :) thanks Matthias!
>
> Tomorrow I will spend some time to look through your work and give some
> comments.
>
> Also I would love to help with this effort so once we merge an initial
> prototype let's open some Jiras and I will pick some up :)
>
> Gyula
>
> On Thursday, April 2, 2015, Márton Balassi <[hidden email]> wrote:
>
>> Hey Mathias,
>>
>> Thanks, this is a really nice contribution. I just scrolled through the
>> code, but I really like it and big thanks for the the tests for the
>> examples.
>>
>> The rebase Fabian suggested would help a lot when merging.
>>
>>
>>
>> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]
>> <javascript:;>> wrote:
>>
>>> Hi Matthias,
>>>
>>> this is really cool!I especially like that you can use Storm code within
>> a
>>> Flink streaming program :-)
>>>
>>> One thing that might be good to do rather soon is to collect all your
>>> commits and put them on top of a fresh forked Flink master branch.
>>> When merging we cannot change the history and try to put only
>> fast-forward
>>> commits on top of the master branch.
>>> As time goes on it becomes more likely that you run into merge issues
>> when
>>> cherry-picking the commits.
>>>
>>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]
>> <javascript:;>>:
>>>
>>>> Hey Henry,
>>>>
>>>> you can check out the files here:
>>>>
>>>>
>>>
>> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
>>>> ... so yes, they are located in the flink-streaming directory .. which
>>> is a
>>>> good place for now.
>>>> Once we move flink-streaming out of staging, we might want to keep the
>>>> storm compat in staging.
>>>>
>>>>
>>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <[hidden email]
>> <javascript:;>>
>>>> wrote:
>>>>
>>>>> HI Matthias,
>>>>>
>>>>> Where do you put the code for the Storm compatibility? Under streams
>>>>> module directory?
>>>>>
>>>>> - Henry
>>>>>
>>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
>>>>> <[hidden email] <javascript:;>> wrote:
>>>>>> Hi @all,
>>>>>>
>>>>>> I started to work on an compatibility layer to run Storm Topologies
>>> on
>>>>>> Flink. I just pushed a first beta:
>>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
>>>>>>
>>>>>> Please check it out, and let me know how you like it. In this first
>>>>>> version, I tried to code without changing too many things in the
>>>>>> existing code. However, a deeper integration would be nice.
>>>>>>
>>>>>> There are many things that do not work yet. Currently, only shuffle
>>> and
>>>>>> fieldsGrouping is supported (and only Storm's default output
>> stream).
>>>>>> Furthermore, topologies must be "simple", ie, they cannot be
>>> configured
>>>>>> with a Config object and Storm meta information (ie,
>> TopologyContext)
>>>> is
>>>>>> also not there.
>>>>>>
>>>>>> The layer can be used to integrate existing Spouts and/or Bolts as
>>>> Flink
>>>>>> operators into a regular Flink program. Furthermore, a whole Storm
>>>>>> topology can be executed by switching from TopologyBuilder to
>>>>>> FlinkTopologyBuilder and from
>>> LocalCluster/NimbusClient/StormSubmitter
>>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
>>>>>>
>>>>>> Examples of both cases are given as ITCases.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>  Matthias
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Fabian Hueske-2
Right now, your commits in your working branch are mixed with commits which
are already in pushed to the master branch.
Merging this branch to the master branch in order to push it to our master
might turn out into a complex merging process.
Merging becomes far easier for us, if all commits that you add are placed
on top of the latest Flink master branch.

To setup a working branch, I would start with the latest Flink master
branch and cherry-pick your commits from first to last to that branch. If
you want to update the Flink master branch, pull the changes to the master
branch and rebase your working branch onto the master branch. That way your
commit will be placed on top again.

2015-04-03 0:03 GMT+02:00 Paris Carbone <[hidden email]>:

> That’s pretty nice Matthias, we could use a compositional API in streaming
> that many people are familiar with.
> I can also help in some parts, I see some issues we already encountered
> while creating the samoa adapter (eg. dealing with circles in the
> topology). Thanks again for initiating this!
>
> Paris
>
> > On 02 Apr 2015, at 23:14, Gyula Fóra <[hidden email]> wrote:
> >
> > This sounds amazing :) thanks Matthias!
> >
> > Tomorrow I will spend some time to look through your work and give some
> > comments.
> >
> > Also I would love to help with this effort so once we merge an initial
> > prototype let's open some Jiras and I will pick some up :)
> >
> > Gyula
> >
> > On Thursday, April 2, 2015, Márton Balassi <[hidden email]>
> wrote:
> >
> >> Hey Mathias,
> >>
> >> Thanks, this is a really nice contribution. I just scrolled through the
> >> code, but I really like it and big thanks for the the tests for the
> >> examples.
> >>
> >> The rebase Fabian suggested would help a lot when merging.
> >>
> >>
> >>
> >> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]
> >> <javascript:;>> wrote:
> >>
> >>> Hi Matthias,
> >>>
> >>> this is really cool!I especially like that you can use Storm code
> within
> >> a
> >>> Flink streaming program :-)
> >>>
> >>> One thing that might be good to do rather soon is to collect all your
> >>> commits and put them on top of a fresh forked Flink master branch.
> >>> When merging we cannot change the history and try to put only
> >> fast-forward
> >>> commits on top of the master branch.
> >>> As time goes on it becomes more likely that you run into merge issues
> >> when
> >>> cherry-picking the commits.
> >>>
> >>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]
> >> <javascript:;>>:
> >>>
> >>>> Hey Henry,
> >>>>
> >>>> you can check out the files here:
> >>>>
> >>>>
> >>>
> >>
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> >>>> ... so yes, they are located in the flink-streaming directory .. which
> >>> is a
> >>>> good place for now.
> >>>> Once we move flink-streaming out of staging, we might want to keep the
> >>>> storm compat in staging.
> >>>>
> >>>>
> >>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <
> [hidden email]
> >> <javascript:;>>
> >>>> wrote:
> >>>>
> >>>>> HI Matthias,
> >>>>>
> >>>>> Where do you put the code for the Storm compatibility? Under streams
> >>>>> module directory?
> >>>>>
> >>>>> - Henry
> >>>>>
> >>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> >>>>> <[hidden email] <javascript:;>> wrote:
> >>>>>> Hi @all,
> >>>>>>
> >>>>>> I started to work on an compatibility layer to run Storm Topologies
> >>> on
> >>>>>> Flink. I just pushed a first beta:
> >>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
> >>>>>>
> >>>>>> Please check it out, and let me know how you like it. In this first
> >>>>>> version, I tried to code without changing too many things in the
> >>>>>> existing code. However, a deeper integration would be nice.
> >>>>>>
> >>>>>> There are many things that do not work yet. Currently, only shuffle
> >>> and
> >>>>>> fieldsGrouping is supported (and only Storm's default output
> >> stream).
> >>>>>> Furthermore, topologies must be "simple", ie, they cannot be
> >>> configured
> >>>>>> with a Config object and Storm meta information (ie,
> >> TopologyContext)
> >>>> is
> >>>>>> also not there.
> >>>>>>
> >>>>>> The layer can be used to integrate existing Spouts and/or Bolts as
> >>>> Flink
> >>>>>> operators into a regular Flink program. Furthermore, a whole Storm
> >>>>>> topology can be executed by switching from TopologyBuilder to
> >>>>>> FlinkTopologyBuilder and from
> >>> LocalCluster/NimbusClient/StormSubmitter
> >>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> >>>>>>
> >>>>>> Examples of both cases are given as ITCases.
> >>>>>>
> >>>>>>
> >>>>>> Cheers,
> >>>>>>  Matthias
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Robert Metzger
As far as I understood git rebase [1], cherry-picking all changes in order
to the current master is exactly equal to "git rebase flink/master".
The problem is that you have to resolve all conflicts again. But in this
case the changes to existing code are pretty small, so that might actually
work conflict-free.


[1] http://git-scm.com/docs/git-rebase

On Fri, Apr 3, 2015 at 9:13 PM, Fabian Hueske <[hidden email]> wrote:

> Right now, your commits in your working branch are mixed with commits which
> are already in pushed to the master branch.
> Merging this branch to the master branch in order to push it to our master
> might turn out into a complex merging process.
> Merging becomes far easier for us, if all commits that you add are placed
> on top of the latest Flink master branch.
>
> To setup a working branch, I would start with the latest Flink master
> branch and cherry-pick your commits from first to last to that branch. If
> you want to update the Flink master branch, pull the changes to the master
> branch and rebase your working branch onto the master branch. That way your
> commit will be placed on top again.
>
> 2015-04-03 0:03 GMT+02:00 Paris Carbone <[hidden email]>:
>
> > That’s pretty nice Matthias, we could use a compositional API in
> streaming
> > that many people are familiar with.
> > I can also help in some parts, I see some issues we already encountered
> > while creating the samoa adapter (eg. dealing with circles in the
> > topology). Thanks again for initiating this!
> >
> > Paris
> >
> > > On 02 Apr 2015, at 23:14, Gyula Fóra <[hidden email]> wrote:
> > >
> > > This sounds amazing :) thanks Matthias!
> > >
> > > Tomorrow I will spend some time to look through your work and give some
> > > comments.
> > >
> > > Also I would love to help with this effort so once we merge an initial
> > > prototype let's open some Jiras and I will pick some up :)
> > >
> > > Gyula
> > >
> > > On Thursday, April 2, 2015, Márton Balassi <[hidden email]>
> > wrote:
> > >
> > >> Hey Mathias,
> > >>
> > >> Thanks, this is a really nice contribution. I just scrolled through
> the
> > >> code, but I really like it and big thanks for the the tests for the
> > >> examples.
> > >>
> > >> The rebase Fabian suggested would help a lot when merging.
> > >>
> > >>
> > >>
> > >> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]
> > >> <javascript:;>> wrote:
> > >>
> > >>> Hi Matthias,
> > >>>
> > >>> this is really cool!I especially like that you can use Storm code
> > within
> > >> a
> > >>> Flink streaming program :-)
> > >>>
> > >>> One thing that might be good to do rather soon is to collect all your
> > >>> commits and put them on top of a fresh forked Flink master branch.
> > >>> When merging we cannot change the history and try to put only
> > >> fast-forward
> > >>> commits on top of the master branch.
> > >>> As time goes on it becomes more likely that you run into merge issues
> > >> when
> > >>> cherry-picking the commits.
> > >>>
> > >>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]
> > >> <javascript:;>>:
> > >>>
> > >>>> Hey Henry,
> > >>>>
> > >>>> you can check out the files here:
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> > >>>> ... so yes, they are located in the flink-streaming directory ..
> which
> > >>> is a
> > >>>> good place for now.
> > >>>> Once we move flink-streaming out of staging, we might want to keep
> the
> > >>>> storm compat in staging.
> > >>>>
> > >>>>
> > >>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <
> > [hidden email]
> > >> <javascript:;>>
> > >>>> wrote:
> > >>>>
> > >>>>> HI Matthias,
> > >>>>>
> > >>>>> Where do you put the code for the Storm compatibility? Under
> streams
> > >>>>> module directory?
> > >>>>>
> > >>>>> - Henry
> > >>>>>
> > >>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> > >>>>> <[hidden email] <javascript:;>> wrote:
> > >>>>>> Hi @all,
> > >>>>>>
> > >>>>>> I started to work on an compatibility layer to run Storm
> Topologies
> > >>> on
> > >>>>>> Flink. I just pushed a first beta:
> > >>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
> > >>>>>>
> > >>>>>> Please check it out, and let me know how you like it. In this
> first
> > >>>>>> version, I tried to code without changing too many things in the
> > >>>>>> existing code. However, a deeper integration would be nice.
> > >>>>>>
> > >>>>>> There are many things that do not work yet. Currently, only
> shuffle
> > >>> and
> > >>>>>> fieldsGrouping is supported (and only Storm's default output
> > >> stream).
> > >>>>>> Furthermore, topologies must be "simple", ie, they cannot be
> > >>> configured
> > >>>>>> with a Config object and Storm meta information (ie,
> > >> TopologyContext)
> > >>>> is
> > >>>>>> also not there.
> > >>>>>>
> > >>>>>> The layer can be used to integrate existing Spouts and/or Bolts as
> > >>>> Flink
> > >>>>>> operators into a regular Flink program. Furthermore, a whole Storm
> > >>>>>> topology can be executed by switching from TopologyBuilder to
> > >>>>>> FlinkTopologyBuilder and from
> > >>> LocalCluster/NimbusClient/StormSubmitter
> > >>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> > >>>>>>
> > >>>>>> Examples of both cases are given as ITCases.
> > >>>>>>
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>>  Matthias
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Matthias J. Sax
Done. Shall I open a pull request?

-Matthias


On 04/03/2015 09:32 PM, Robert Metzger wrote:

> As far as I understood git rebase [1], cherry-picking all changes in order
> to the current master is exactly equal to "git rebase flink/master".
> The problem is that you have to resolve all conflicts again. But in this
> case the changes to existing code are pretty small, so that might actually
> work conflict-free.
>
>
> [1] http://git-scm.com/docs/git-rebase
>
> On Fri, Apr 3, 2015 at 9:13 PM, Fabian Hueske <[hidden email]> wrote:
>
>> Right now, your commits in your working branch are mixed with commits which
>> are already in pushed to the master branch.
>> Merging this branch to the master branch in order to push it to our master
>> might turn out into a complex merging process.
>> Merging becomes far easier for us, if all commits that you add are placed
>> on top of the latest Flink master branch.
>>
>> To setup a working branch, I would start with the latest Flink master
>> branch and cherry-pick your commits from first to last to that branch. If
>> you want to update the Flink master branch, pull the changes to the master
>> branch and rebase your working branch onto the master branch. That way your
>> commit will be placed on top again.
>>
>> 2015-04-03 0:03 GMT+02:00 Paris Carbone <[hidden email]>:
>>
>>> That’s pretty nice Matthias, we could use a compositional API in
>> streaming
>>> that many people are familiar with.
>>> I can also help in some parts, I see some issues we already encountered
>>> while creating the samoa adapter (eg. dealing with circles in the
>>> topology). Thanks again for initiating this!
>>>
>>> Paris
>>>
>>>> On 02 Apr 2015, at 23:14, Gyula Fóra <[hidden email]> wrote:
>>>>
>>>> This sounds amazing :) thanks Matthias!
>>>>
>>>> Tomorrow I will spend some time to look through your work and give some
>>>> comments.
>>>>
>>>> Also I would love to help with this effort so once we merge an initial
>>>> prototype let's open some Jiras and I will pick some up :)
>>>>
>>>> Gyula
>>>>
>>>> On Thursday, April 2, 2015, Márton Balassi <[hidden email]>
>>> wrote:
>>>>
>>>>> Hey Mathias,
>>>>>
>>>>> Thanks, this is a really nice contribution. I just scrolled through
>> the
>>>>> code, but I really like it and big thanks for the the tests for the
>>>>> examples.
>>>>>
>>>>> The rebase Fabian suggested would help a lot when merging.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]
>>>>> <javascript:;>> wrote:
>>>>>
>>>>>> Hi Matthias,
>>>>>>
>>>>>> this is really cool!I especially like that you can use Storm code
>>> within
>>>>> a
>>>>>> Flink streaming program :-)
>>>>>>
>>>>>> One thing that might be good to do rather soon is to collect all your
>>>>>> commits and put them on top of a fresh forked Flink master branch.
>>>>>> When merging we cannot change the history and try to put only
>>>>> fast-forward
>>>>>> commits on top of the master branch.
>>>>>> As time goes on it becomes more likely that you run into merge issues
>>>>> when
>>>>>> cherry-picking the commits.
>>>>>>
>>>>>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]
>>>>> <javascript:;>>:
>>>>>>
>>>>>>> Hey Henry,
>>>>>>>
>>>>>>> you can check out the files here:
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
>>>>>>> ... so yes, they are located in the flink-streaming directory ..
>> which
>>>>>> is a
>>>>>>> good place for now.
>>>>>>> Once we move flink-streaming out of staging, we might want to keep
>> the
>>>>>>> storm compat in staging.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <
>>> [hidden email]
>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> HI Matthias,
>>>>>>>>
>>>>>>>> Where do you put the code for the Storm compatibility? Under
>> streams
>>>>>>>> module directory?
>>>>>>>>
>>>>>>>> - Henry
>>>>>>>>
>>>>>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
>>>>>>>> <[hidden email] <javascript:;>> wrote:
>>>>>>>>> Hi @all,
>>>>>>>>>
>>>>>>>>> I started to work on an compatibility layer to run Storm
>> Topologies
>>>>>> on
>>>>>>>>> Flink. I just pushed a first beta:
>>>>>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
>>>>>>>>>
>>>>>>>>> Please check it out, and let me know how you like it. In this
>> first
>>>>>>>>> version, I tried to code without changing too many things in the
>>>>>>>>> existing code. However, a deeper integration would be nice.
>>>>>>>>>
>>>>>>>>> There are many things that do not work yet. Currently, only
>> shuffle
>>>>>> and
>>>>>>>>> fieldsGrouping is supported (and only Storm's default output
>>>>> stream).
>>>>>>>>> Furthermore, topologies must be "simple", ie, they cannot be
>>>>>> configured
>>>>>>>>> with a Config object and Storm meta information (ie,
>>>>> TopologyContext)
>>>>>>> is
>>>>>>>>> also not there.
>>>>>>>>>
>>>>>>>>> The layer can be used to integrate existing Spouts and/or Bolts as
>>>>>>> Flink
>>>>>>>>> operators into a regular Flink program. Furthermore, a whole Storm
>>>>>>>>> topology can be executed by switching from TopologyBuilder to
>>>>>>>>> FlinkTopologyBuilder and from
>>>>>> LocalCluster/NimbusClient/StormSubmitter
>>>>>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
>>>>>>>>>
>>>>>>>>> Examples of both cases are given as ITCases.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>  Matthias
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Storm compatibility layer for Flink (first beta available)

Márton Balassi
Thanks, Matthias. Yes, please. :)

On Mon, Apr 6, 2015 at 3:40 PM, Matthias J. Sax <
[hidden email]> wrote:

> Done. Shall I open a pull request?
>
> -Matthias
>
>
> On 04/03/2015 09:32 PM, Robert Metzger wrote:
> > As far as I understood git rebase [1], cherry-picking all changes in
> order
> > to the current master is exactly equal to "git rebase flink/master".
> > The problem is that you have to resolve all conflicts again. But in this
> > case the changes to existing code are pretty small, so that might
> actually
> > work conflict-free.
> >
> >
> > [1] http://git-scm.com/docs/git-rebase
> >
> > On Fri, Apr 3, 2015 at 9:13 PM, Fabian Hueske <[hidden email]> wrote:
> >
> >> Right now, your commits in your working branch are mixed with commits
> which
> >> are already in pushed to the master branch.
> >> Merging this branch to the master branch in order to push it to our
> master
> >> might turn out into a complex merging process.
> >> Merging becomes far easier for us, if all commits that you add are
> placed
> >> on top of the latest Flink master branch.
> >>
> >> To setup a working branch, I would start with the latest Flink master
> >> branch and cherry-pick your commits from first to last to that branch.
> If
> >> you want to update the Flink master branch, pull the changes to the
> master
> >> branch and rebase your working branch onto the master branch. That way
> your
> >> commit will be placed on top again.
> >>
> >> 2015-04-03 0:03 GMT+02:00 Paris Carbone <[hidden email]>:
> >>
> >>> That’s pretty nice Matthias, we could use a compositional API in
> >> streaming
> >>> that many people are familiar with.
> >>> I can also help in some parts, I see some issues we already encountered
> >>> while creating the samoa adapter (eg. dealing with circles in the
> >>> topology). Thanks again for initiating this!
> >>>
> >>> Paris
> >>>
> >>>> On 02 Apr 2015, at 23:14, Gyula Fóra <[hidden email]> wrote:
> >>>>
> >>>> This sounds amazing :) thanks Matthias!
> >>>>
> >>>> Tomorrow I will spend some time to look through your work and give
> some
> >>>> comments.
> >>>>
> >>>> Also I would love to help with this effort so once we merge an initial
> >>>> prototype let's open some Jiras and I will pick some up :)
> >>>>
> >>>> Gyula
> >>>>
> >>>> On Thursday, April 2, 2015, Márton Balassi <[hidden email]>
> >>> wrote:
> >>>>
> >>>>> Hey Mathias,
> >>>>>
> >>>>> Thanks, this is a really nice contribution. I just scrolled through
> >> the
> >>>>> code, but I really like it and big thanks for the the tests for the
> >>>>> examples.
> >>>>>
> >>>>> The rebase Fabian suggested would help a lot when merging.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Apr 2, 2015 at 9:19 PM, Fabian Hueske <[hidden email]
> >>>>> <javascript:;>> wrote:
> >>>>>
> >>>>>> Hi Matthias,
> >>>>>>
> >>>>>> this is really cool!I especially like that you can use Storm code
> >>> within
> >>>>> a
> >>>>>> Flink streaming program :-)
> >>>>>>
> >>>>>> One thing that might be good to do rather soon is to collect all
> your
> >>>>>> commits and put them on top of a fresh forked Flink master branch.
> >>>>>> When merging we cannot change the history and try to put only
> >>>>> fast-forward
> >>>>>> commits on top of the master branch.
> >>>>>> As time goes on it becomes more likely that you run into merge
> issues
> >>>>> when
> >>>>>> cherry-picking the commits.
> >>>>>>
> >>>>>> 2015-04-02 21:09 GMT+02:00 Robert Metzger <[hidden email]
> >>>>> <javascript:;>>:
> >>>>>>
> >>>>>>> Hey Henry,
> >>>>>>>
> >>>>>>> you can check out the files here:
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/mjsax/flink/tree/flink-storm-compatibility/flink-staging/flink-streaming/flink-storm-compatibility
> >>>>>>> ... so yes, they are located in the flink-streaming directory ..
> >> which
> >>>>>> is a
> >>>>>>> good place for now.
> >>>>>>> Once we move flink-streaming out of staging, we might want to keep
> >> the
> >>>>>>> storm compat in staging.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Apr 2, 2015 at 8:16 PM, Henry Saputra <
> >>> [hidden email]
> >>>>> <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> HI Matthias,
> >>>>>>>>
> >>>>>>>> Where do you put the code for the Storm compatibility? Under
> >> streams
> >>>>>>>> module directory?
> >>>>>>>>
> >>>>>>>> - Henry
> >>>>>>>>
> >>>>>>>> On Thu, Apr 2, 2015 at 10:31 AM, Matthias J. Sax
> >>>>>>>> <[hidden email] <javascript:;>> wrote:
> >>>>>>>>> Hi @all,
> >>>>>>>>>
> >>>>>>>>> I started to work on an compatibility layer to run Storm
> >> Topologies
> >>>>>> on
> >>>>>>>>> Flink. I just pushed a first beta:
> >>>>>>>>> https://github.com/mjsax/flink/tree/flink-storm-compatibility
> >>>>>>>>>
> >>>>>>>>> Please check it out, and let me know how you like it. In this
> >> first
> >>>>>>>>> version, I tried to code without changing too many things in the
> >>>>>>>>> existing code. However, a deeper integration would be nice.
> >>>>>>>>>
> >>>>>>>>> There are many things that do not work yet. Currently, only
> >> shuffle
> >>>>>> and
> >>>>>>>>> fieldsGrouping is supported (and only Storm's default output
> >>>>> stream).
> >>>>>>>>> Furthermore, topologies must be "simple", ie, they cannot be
> >>>>>> configured
> >>>>>>>>> with a Config object and Storm meta information (ie,
> >>>>> TopologyContext)
> >>>>>>> is
> >>>>>>>>> also not there.
> >>>>>>>>>
> >>>>>>>>> The layer can be used to integrate existing Spouts and/or Bolts
> as
> >>>>>>> Flink
> >>>>>>>>> operators into a regular Flink program. Furthermore, a whole
> Storm
> >>>>>>>>> topology can be executed by switching from TopologyBuilder to
> >>>>>>>>> FlinkTopologyBuilder and from
> >>>>>> LocalCluster/NimbusClient/StormSubmitter
> >>>>>>>>> to FlinkLocalCluster/FlinkClient/FlinkStormSubmitter.
> >>>>>>>>>
> >>>>>>>>> Examples of both cases are given as ITCases.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>  Matthias
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
> >
>
>