The purpose of flink-contrib currently is to hold contributions to the
project that we do not consider part of the core flink functionality, but provide useful tools around it. In general code placed here has to meet less requirements in terms of covering all corner cases if it provides a nice solution for a set of well defined problems. As of today it has two small utilities, the TweetInputFormat (by Mustafa Elbehery) and the collect functionality for the DataStream (by Gabor Gevay). The pull request for the Storm compatibility layer (by Matthias J. Sax) [1] raises the issue as it is way more code to maintain and is more complex in general that how the community would like to handle these in terms of distribution. Do we want to have it in the Flink repository or maybe in a separate one. I am personally really for having the Storm compatibility layer under flink-contrib as Matthias is very active on the mailing list and has also expressed his interest of further developing the functionality of the compatibility layer. To top that a couple of users got excited about the new feature, so I see no risk in having this code in the main repository. As for the structure of flink-contrib I would have the contents separated to slim as possible maven projects, to make sure that the users only get the dependencies that they really need. [1] https://github.com/apache/flink/pull/573 |
Hi,
Thank you for starting the discussion Marton! I would really like to merge the storm compat to our source repo. I think that code which is not merged there will not get enough attention. I'm against splitting flink-contrib into small maven modules. I totally understand your reasoning (mixed dependencies), but "flink-staging" exists exactly for that purpose (one maven module per "beta"-module). For now, users depending on flink-contrib have to define exclusions to control the dependencies. So I'm +1 for merging it to "flink-contrib". I guess not all committers have time to look into the pull request, therefore, I want to remind you that the code contains a lot of TODOs and UnsupportedOperationExceptions. @Marton: Do you know anybody from the Budapest Flink Streaming crew with some Storm experience who could try out the code on a cluster and give some feedback? On Tue, May 12, 2015 at 9:52 AM, Márton Balassi <[hidden email]> wrote: > The purpose of flink-contrib currently is to hold contributions to the > project that we do not consider part of the core flink functionality, but > provide useful tools around it. In general code placed here has to meet > less requirements in terms of covering all corner cases if it provides a > nice solution for a set of well defined problems. > > As of today it has two small utilities, the TweetInputFormat (by Mustafa > Elbehery) and the collect functionality for the DataStream (by Gabor > Gevay). > > The pull request for the Storm compatibility layer (by Matthias J. Sax) [1] > raises the issue as it is way more code to maintain and is more complex in > general that how the community would like to handle these in terms of > distribution. Do we want to have it in the Flink repository or maybe in a > separate one. > > I am personally really for having the Storm compatibility layer under > flink-contrib as Matthias is very active on the mailing list and has also > expressed his interest of further developing the functionality of the > compatibility layer. To top that a couple of users got excited about the > new feature, so I see no risk in having this code in the main repository. > > As for the structure of flink-contrib I would have the contents separated > to slim as possible maven projects, to make sure that the users only get > the dependencies that they really need. > > [1] https://github.com/apache/flink/pull/573 > |
In reply to this post by Márton Balassi
+1 to merge it to contrib.
We should hold a path open that - inactive code gets dropped from contrib after a while - very active code may gets its own project (inside or outside the Flink codebase, whatever fits better) On Tue, May 12, 2015 at 9:52 AM, Márton Balassi <[hidden email]> wrote: > The purpose of flink-contrib currently is to hold contributions to the > project that we do not consider part of the core flink functionality, but > provide useful tools around it. In general code placed here has to meet > less requirements in terms of covering all corner cases if it provides a > nice solution for a set of well defined problems. > > As of today it has two small utilities, the TweetInputFormat (by Mustafa > Elbehery) and the collect functionality for the DataStream (by Gabor > Gevay). > > The pull request for the Storm compatibility layer (by Matthias J. Sax) [1] > raises the issue as it is way more code to maintain and is more complex in > general that how the community would like to handle these in terms of > distribution. Do we want to have it in the Flink repository or maybe in a > separate one. > > I am personally really for having the Storm compatibility layer under > flink-contrib as Matthias is very active on the mailing list and has also > expressed his interest of further developing the functionality of the > compatibility layer. To top that a couple of users got excited about the > new feature, so I see no risk in having this code in the main repository. > > As for the structure of flink-contrib I would have the contents separated > to slim as possible maven projects, to make sure that the users only get > the dependencies that they really need. > > [1] https://github.com/apache/flink/pull/573 > |
In reply to this post by Robert Metzger
Hi,
some UnsupportedOperationExceptions are required, because Storm interfaces are implement but Flink cannot support those functionality. Some other are "not yet implemented" once. A few other of them could be removed (in case an interface in not implemented, but only mimicked), by removing the whole method. I prefer to mimic interfaced completely and through UnsupportedOperationExceptions, because if a user wants to execute Storm code on Flink, less changes to the original Storm code are necessary, making the transition easier. Of course, it is a valid argument to remove the methods completely to raise incompatibilities directly are compile time. The TODOs are points that might or might not be Flink compatible. This must be checked and maybe discussed. Any feedback is welcome. -Matthias On 05/12/2015 10:46 AM, Robert Metzger wrote: > Hi, > > Thank you for starting the discussion Marton! > > I would really like to merge the storm compat to our source repo. I think > that code which is not merged there will not get enough attention. > > I'm against splitting flink-contrib into small maven modules. I totally > understand your reasoning (mixed dependencies), but "flink-staging" exists > exactly for that purpose (one maven module per "beta"-module). > For now, users depending on flink-contrib have to define exclusions to > control the dependencies. > > So I'm +1 for merging it to "flink-contrib". > I guess not all committers have time to look into the pull request, > therefore, I want to remind you that the code contains a lot of TODOs and > UnsupportedOperationExceptions. > > @Marton: Do you know anybody from the Budapest Flink Streaming crew with > some Storm experience who could try out the code on a cluster and give some > feedback? > > > On Tue, May 12, 2015 at 9:52 AM, Márton Balassi <[hidden email]> > wrote: > >> The purpose of flink-contrib currently is to hold contributions to the >> project that we do not consider part of the core flink functionality, but >> provide useful tools around it. In general code placed here has to meet >> less requirements in terms of covering all corner cases if it provides a >> nice solution for a set of well defined problems. >> >> As of today it has two small utilities, the TweetInputFormat (by Mustafa >> Elbehery) and the collect functionality for the DataStream (by Gabor >> Gevay). >> >> The pull request for the Storm compatibility layer (by Matthias J. Sax) [1] >> raises the issue as it is way more code to maintain and is more complex in >> general that how the community would like to handle these in terms of >> distribution. Do we want to have it in the Flink repository or maybe in a >> separate one. >> >> I am personally really for having the Storm compatibility layer under >> flink-contrib as Matthias is very active on the mailing list and has also >> expressed his interest of further developing the functionality of the >> compatibility layer. To top that a couple of users got excited about the >> new feature, so I see no risk in having this code in the main repository. >> >> As for the structure of flink-contrib I would have the contents separated >> to slim as possible maven projects, to make sure that the users only get >> the dependencies that they really need. >> >> [1] https://github.com/apache/flink/pull/573 >> > |
@Robert: I have a little storm experience. I will try to run some examples
on the cluster. Peter 2015-05-12 11:40 GMT+02:00 Matthias J. Sax <[hidden email]>: > Hi, > > some UnsupportedOperationExceptions are required, because Storm > interfaces are implement but Flink cannot support those functionality. > Some other are "not yet implemented" once. > > A few other of them could be removed (in case an interface in not > implemented, but only mimicked), by removing the whole method. I prefer > to mimic interfaced completely and through > UnsupportedOperationExceptions, because if a user wants to execute Storm > code on Flink, less changes to the original Storm code are necessary, > making the transition easier. > > Of course, it is a valid argument to remove the methods completely to > raise incompatibilities directly are compile time. > > The TODOs are points that might or might not be Flink compatible. This > must be checked and maybe discussed. > > Any feedback is welcome. > > > -Matthias > > On 05/12/2015 10:46 AM, Robert Metzger wrote: > > Hi, > > > > Thank you for starting the discussion Marton! > > > > I would really like to merge the storm compat to our source repo. I think > > that code which is not merged there will not get enough attention. > > > > I'm against splitting flink-contrib into small maven modules. I totally > > understand your reasoning (mixed dependencies), but "flink-staging" > exists > > exactly for that purpose (one maven module per "beta"-module). > > For now, users depending on flink-contrib have to define exclusions to > > control the dependencies. > > > > So I'm +1 for merging it to "flink-contrib". > > I guess not all committers have time to look into the pull request, > > therefore, I want to remind you that the code contains a lot of TODOs and > > UnsupportedOperationExceptions. > > > > @Marton: Do you know anybody from the Budapest Flink Streaming crew with > > some Storm experience who could try out the code on a cluster and give > some > > feedback? > > > > > > On Tue, May 12, 2015 at 9:52 AM, Márton Balassi < > [hidden email]> > > wrote: > > > >> The purpose of flink-contrib currently is to hold contributions to the > >> project that we do not consider part of the core flink functionality, > but > >> provide useful tools around it. In general code placed here has to meet > >> less requirements in terms of covering all corner cases if it provides a > >> nice solution for a set of well defined problems. > >> > >> As of today it has two small utilities, the TweetInputFormat (by Mustafa > >> Elbehery) and the collect functionality for the DataStream (by Gabor > >> Gevay). > >> > >> The pull request for the Storm compatibility layer (by Matthias J. Sax) > [1] > >> raises the issue as it is way more code to maintain and is more complex > in > >> general that how the community would like to handle these in terms of > >> distribution. Do we want to have it in the Flink repository or maybe in > a > >> separate one. > >> > >> I am personally really for having the Storm compatibility layer under > >> flink-contrib as Matthias is very active on the mailing list and has > also > >> expressed his interest of further developing the functionality of the > >> compatibility layer. To top that a couple of users got excited about the > >> new feature, so I see no risk in having this code in the main > repository. > >> > >> As for the structure of flink-contrib I would have the contents > separated > >> to slim as possible maven projects, to make sure that the users only get > >> the dependencies that they really need. > >> > >> [1] https://github.com/apache/flink/pull/573 > >> > > > > |
That would be great, thank you!
Please write down everything you see and post it on the mailing list. On Wed, May 13, 2015 at 9:42 AM, Szabó Péter <[hidden email]> wrote: > @Robert: I have a little storm experience. I will try to run some examples > on the cluster. > > Peter > > 2015-05-12 11:40 GMT+02:00 Matthias J. Sax <[hidden email] > >: > > > Hi, > > > > some UnsupportedOperationExceptions are required, because Storm > > interfaces are implement but Flink cannot support those functionality. > > Some other are "not yet implemented" once. > > > > A few other of them could be removed (in case an interface in not > > implemented, but only mimicked), by removing the whole method. I prefer > > to mimic interfaced completely and through > > UnsupportedOperationExceptions, because if a user wants to execute Storm > > code on Flink, less changes to the original Storm code are necessary, > > making the transition easier. > > > > Of course, it is a valid argument to remove the methods completely to > > raise incompatibilities directly are compile time. > > > > The TODOs are points that might or might not be Flink compatible. This > > must be checked and maybe discussed. > > > > Any feedback is welcome. > > > > > > -Matthias > > > > On 05/12/2015 10:46 AM, Robert Metzger wrote: > > > Hi, > > > > > > Thank you for starting the discussion Marton! > > > > > > I would really like to merge the storm compat to our source repo. I > think > > > that code which is not merged there will not get enough attention. > > > > > > I'm against splitting flink-contrib into small maven modules. I totally > > > understand your reasoning (mixed dependencies), but "flink-staging" > > exists > > > exactly for that purpose (one maven module per "beta"-module). > > > For now, users depending on flink-contrib have to define exclusions to > > > control the dependencies. > > > > > > So I'm +1 for merging it to "flink-contrib". > > > I guess not all committers have time to look into the pull request, > > > therefore, I want to remind you that the code contains a lot of TODOs > and > > > UnsupportedOperationExceptions. > > > > > > @Marton: Do you know anybody from the Budapest Flink Streaming crew > with > > > some Storm experience who could try out the code on a cluster and give > > some > > > feedback? > > > > > > > > > On Tue, May 12, 2015 at 9:52 AM, Márton Balassi < > > [hidden email]> > > > wrote: > > > > > >> The purpose of flink-contrib currently is to hold contributions to the > > >> project that we do not consider part of the core flink functionality, > > but > > >> provide useful tools around it. In general code placed here has to > meet > > >> less requirements in terms of covering all corner cases if it > provides a > > >> nice solution for a set of well defined problems. > > >> > > >> As of today it has two small utilities, the TweetInputFormat (by > Mustafa > > >> Elbehery) and the collect functionality for the DataStream (by Gabor > > >> Gevay). > > >> > > >> The pull request for the Storm compatibility layer (by Matthias J. > Sax) > > [1] > > >> raises the issue as it is way more code to maintain and is more > complex > > in > > >> general that how the community would like to handle these in terms of > > >> distribution. Do we want to have it in the Flink repository or maybe > in > > a > > >> separate one. > > >> > > >> I am personally really for having the Storm compatibility layer under > > >> flink-contrib as Matthias is very active on the mailing list and has > > also > > >> expressed his interest of further developing the functionality of the > > >> compatibility layer. To top that a couple of users got excited about > the > > >> new feature, so I see no risk in having this code in the main > > repository. > > >> > > >> As for the structure of flink-contrib I would have the contents > > separated > > >> to slim as possible maven projects, to make sure that the users only > get > > >> the dependencies that they really need. > > >> > > >> [1] https://github.com/apache/flink/pull/573 > > >> > > > > > > > > |
Free forum by Nabble | Edit this page |