Adding non-core API features to Flink

[Fabian Hueske-2]

Hi all,

we got a few contribution requests lately to add cool but "non-core"
features to our API.
In previous discussions, concerns were raised to not bloat the APIs with
too many "shortcut", "syntactic sugar", or special-case features.

Instead we could setup a place to add Input/OutputFormats, common
operations, etc. which does not need as much control as the core APIs. Open
questions are:
- How do we organize it? (top-level maven module, modules in flink-java,
flink-scala, java packages in the API modules, ...)
- How do we name it? flink-utils, flink-packages, ...

Any opinions on this?

Cheers, Fabian

[Chiwan Park]

I think top level maven module called "flink-contrib" is reasonable. There are other projects having contrib package such as Akka, Django.

Regards, Chiwan Park (Sent with iPhone)

2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:

> Hi all,
>
> we got a few contribution requests lately to add cool but "non-core"
> features to our API.
> In previous discussions, concerns were raised to not bloat the APIs with
> too many "shortcut", "syntactic sugar", or special-case features.
>
> Instead we could setup a place to add Input/OutputFormats, common
> operations, etc. which does not need as much control as the core APIs. Open
> questions are:
> - How do we organize it? (top-level maven module, modules in flink-java,
> flink-scala, java packages in the API modules, ...)
> - How do we name it? flink-utils, flink-packages, ...
>
> Any opinions on this?
>
> Cheers, Fabian

[Stephan Ewen]

Yes, a "flink-contrib" project would be great.

We have two options:

1) Make it part of the flink apache project.
  - PRO this makes it easy to get stuff for users
  - CONTRA this means stronger requirements on the code, blocker for code
that uses dependencies under certain licenses, etc.

2) Make an independent github project.
 - PRO contributions can depended on more licenses, such as LGPL
 - PRO we can have more people that commit to this repo, committers can be
different from flink committers
 - CONTRA people need to grab the extensions from a different location


I am slightly biased towards (2), but open to both.

Stephan




On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <[hidden email]> wrote:

> I think top level maven module called "flink-contrib" is reasonable. There
> are other projects having contrib package such as Akka, Django.
>
> Regards, Chiwan Park (Sent with iPhone)
>
> 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
>
> > Hi all,
> >
> > we got a few contribution requests lately to add cool but "non-core"
> > features to our API.
> > In previous discussions, concerns were raised to not bloat the APIs with
> > too many "shortcut", "syntactic sugar", or special-case features.
> >
> > Instead we could setup a place to add Input/OutputFormats, common
> > operations, etc. which does not need as much control as the core APIs.
> Open
> > questions are:
> > - How do we organize it? (top-level maven module, modules in flink-java,
> > flink-scala, java packages in the API modules, ...)
> > - How do we name it? flink-utils, flink-packages, ...
> >
> > Any opinions on this?
> >
> > Cheers, Fabian
>

[Kostas Tzoumas-2]

Thanks Fabian for starting the discussion.

I would be biased towards option (1) that Stephan highlighted for the
following reasons:

- A separate github project is one more infrastructure to manage, and it
lives outside the ASF. I would like to bring as much code as possible to
the Apache Software Foundation, and not divide the codebase into two
separate repositories.

- The personal gratification (and thus motivation) is higher when
contributing to a top-level Apache project than a github repository
slightly associated with an ASF project. And contributors to the Flink
project get karma that may lead to new committers, which is crucial as the
project is growing.

Of course, non Apache-licensed contributions cannot be accepted. If we have
a good amount of those, we can start an infrastructure for Flink packages
that lives outside the ASF, but I would wait for the need to come before
doing this.

My proposal would be to funnel contributions to the main repository (in a
flink-contrib module) for now, including the recent contributions.

Kostas


On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]> wrote:

> Yes, a "flink-contrib" project would be great.
>
> We have two options:
>
> 1) Make it part of the flink apache project.
>   - PRO this makes it easy to get stuff for users
>   - CONTRA this means stronger requirements on the code, blocker for code
> that uses dependencies under certain licenses, etc.
>
> 2) Make an independent github project.
>  - PRO contributions can depended on more licenses, such as LGPL
>  - PRO we can have more people that commit to this repo, committers can be
> different from flink committers
>  - CONTRA people need to grab the extensions from a different location
>
>
> I am slightly biased towards (2), but open to both.
>
> Stephan
>
>
>
>
> On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <[hidden email]>
> wrote:
>
> > I think top level maven module called "flink-contrib" is reasonable.
> There
> > are other projects having contrib package such as Akka, Django.
> >
> > Regards, Chiwan Park (Sent with iPhone)
> >
> > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> >
> > > Hi all,
> > >
> > > we got a few contribution requests lately to add cool but "non-core"
> > > features to our API.
> > > In previous discussions, concerns were raised to not bloat the APIs
> with
> > > too many "shortcut", "syntactic sugar", or special-case features.
> > >
> > > Instead we could setup a place to add Input/OutputFormats, common
> > > operations, etc. which does not need as much control as the core APIs.
> > Open
> > > questions are:
> > > - How do we organize it? (top-level maven module, modules in
> flink-java,
> > > flink-scala, java packages in the API modules, ...)
> > > - How do we name it? flink-utils, flink-packages, ...
> > >
> > > Any opinions on this?
> > >
> > > Cheers, Fabian
> >
>

[Fabian Hueske-2]

I am also more in favor of option 1).

2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:

> Thanks Fabian for starting the discussion.
>
> I would be biased towards option (1) that Stephan highlighted for the
> following reasons:
>
> - A separate github project is one more infrastructure to manage, and it
> lives outside the ASF. I would like to bring as much code as possible to
> the Apache Software Foundation, and not divide the codebase into two
> separate repositories.
>
> - The personal gratification (and thus motivation) is higher when
> contributing to a top-level Apache project than a github repository
> slightly associated with an ASF project. And contributors to the Flink
> project get karma that may lead to new committers, which is crucial as the
> project is growing.
>
> Of course, non Apache-licensed contributions cannot be accepted. If we have
> a good amount of those, we can start an infrastructure for Flink packages
> that lives outside the ASF, but I would wait for the need to come before
> doing this.
>
> My proposal would be to funnel contributions to the main repository (in a
> flink-contrib module) for now, including the recent contributions.
>
> Kostas
>
>
> On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]> wrote:
>
> > Yes, a "flink-contrib" project would be great.
> >
> > We have two options:
> >
> > 1) Make it part of the flink apache project.
> >   - PRO this makes it easy to get stuff for users
> >   - CONTRA this means stronger requirements on the code, blocker for code
> > that uses dependencies under certain licenses, etc.
> >
> > 2) Make an independent github project.
> >  - PRO contributions can depended on more licenses, such as LGPL
> >  - PRO we can have more people that commit to this repo, committers can
> be
> > different from flink committers
> >  - CONTRA people need to grab the extensions from a different location
> >
> >
> > I am slightly biased towards (2), but open to both.
> >
> > Stephan
> >
> >
> >
> >
> > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <[hidden email]>
> > wrote:
> >
> > > I think top level maven module called "flink-contrib" is reasonable.
> > There
> > > are other projects having contrib package such as Akka, Django.
> > >
> > > Regards, Chiwan Park (Sent with iPhone)
> > >
> > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> > >
> > > > Hi all,
> > > >
> > > > we got a few contribution requests lately to add cool but "non-core"
> > > > features to our API.
> > > > In previous discussions, concerns were raised to not bloat the APIs
> > with
> > > > too many "shortcut", "syntactic sugar", or special-case features.
> > > >
> > > > Instead we could setup a place to add Input/OutputFormats, common
> > > > operations, etc. which does not need as much control as the core
> APIs.
> > > Open
> > > > questions are:
> > > > - How do we organize it? (top-level maven module, modules in
> > flink-java,
> > > > flink-scala, java packages in the API modules, ...)
> > > > - How do we name it? flink-utils, flink-packages, ...
> > > >
> > > > Any opinions on this?
> > > >
> > > > Cheers, Fabian
> > >
> >
>

[Ted Dunning]

As the community of flink add-ons grows, a CPAN or maven-like mechanism
might be a nice option.  That would let people download and install
extensions very fluidly.

The argument for making Apache contributions is definitely valid, but the
argument for the agility of fostering independent projects is that projects
can gain lots of popularity very quickly this way.  CPAN, CRAN, pip, maven
and RubyGems can be argued to be critical components of the popularity of
Perl, R, Python, Java/Scala and Ruby respectively.




On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]> wrote:

> I am also more in favor of option 1).
>
> 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
>
> > Thanks Fabian for starting the discussion.
> >
> > I would be biased towards option (1) that Stephan highlighted for the
> > following reasons:
> >
> > - A separate github project is one more infrastructure to manage, and it
> > lives outside the ASF. I would like to bring as much code as possible to
> > the Apache Software Foundation, and not divide the codebase into two
> > separate repositories.
> >
> > - The personal gratification (and thus motivation) is higher when
> > contributing to a top-level Apache project than a github repository
> > slightly associated with an ASF project. And contributors to the Flink
> > project get karma that may lead to new committers, which is crucial as
> the
> > project is growing.
> >
> > Of course, non Apache-licensed contributions cannot be accepted. If we
> have
> > a good amount of those, we can start an infrastructure for Flink packages
> > that lives outside the ASF, but I would wait for the need to come before
> > doing this.
> >
> > My proposal would be to funnel contributions to the main repository (in a
> > flink-contrib module) for now, including the recent contributions.
> >
> > Kostas
> >
> >
> > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Yes, a "flink-contrib" project would be great.
> > >
> > > We have two options:
> > >
> > > 1) Make it part of the flink apache project.
> > >   - PRO this makes it easy to get stuff for users
> > >   - CONTRA this means stronger requirements on the code, blocker for
> code
> > > that uses dependencies under certain licenses, etc.
> > >
> > > 2) Make an independent github project.
> > >  - PRO contributions can depended on more licenses, such as LGPL
> > >  - PRO we can have more people that commit to this repo, committers can
> > be
> > > different from flink committers
> > >  - CONTRA people need to grab the extensions from a different location
> > >
> > >
> > > I am slightly biased towards (2), but open to both.
> > >
> > > Stephan
> > >
> > >
> > >
> > >
> > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <[hidden email]>
> > > wrote:
> > >
> > > > I think top level maven module called "flink-contrib" is reasonable.
> > > There
> > > > are other projects having contrib package such as Akka, Django.
> > > >
> > > > Regards, Chiwan Park (Sent with iPhone)
> > > >
> > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> > > >
> > > > > Hi all,
> > > > >
> > > > > we got a few contribution requests lately to add cool but
> "non-core"
> > > > > features to our API.
> > > > > In previous discussions, concerns were raised to not bloat the APIs
> > > with
> > > > > too many "shortcut", "syntactic sugar", or special-case features.
> > > > >
> > > > > Instead we could setup a place to add Input/OutputFormats, common
> > > > > operations, etc. which does not need as much control as the core
> > APIs.
> > > > Open
> > > > > questions are:
> > > > > - How do we organize it? (top-level maven module, modules in
> > > flink-java,
> > > > > flink-scala, java packages in the API modules, ...)
> > > > > - How do we name it? flink-utils, flink-packages, ...
> > > > >
> > > > > Any opinions on this?
> > > > >
> > > > > Cheers, Fabian
> > > >
> > >
> >
>

[Robert Metzger]

I'm also in favor of option 1) with a "flink-contrib" maven module.

I agree with Ted that we should certainly think about establishing a highly
visible, easy to contribute and easy to use infrastructure for all kinds of
contributions around the project.
But I suspect that we need some time to come up with a good architecture
and infrastructure for that. Maybe this also comes as an outside
contribution to Flink?

To have something immediately, we should start with a "flink-contrib"
module.

One thing that I would like to discuss first is a clear set of rules for
contributions into that module.
Code contributions to "flink-contrib" need:
- to be tested on a cluster (not only by single-jvm tests)
- to have test cases (because otherwise we can not guarantee that they
build with our changes
- to be of use for others
- to have some documentation

I would not deploy the flink-contrib package in the standard flink
distribution. Users will have to add them as a maven dependency.




On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]> wrote:

> As the community of flink add-ons grows, a CPAN or maven-like mechanism
> might be a nice option.  That would let people download and install
> extensions very fluidly.
>
> The argument for making Apache contributions is definitely valid, but the
> argument for the agility of fostering independent projects is that projects
> can gain lots of popularity very quickly this way.  CPAN, CRAN, pip, maven
> and RubyGems can be argued to be critical components of the popularity of
> Perl, R, Python, Java/Scala and Ruby respectively.
>
>
>
>
> On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]> wrote:
>
> > I am also more in favor of option 1).
> >
> > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
> >
> > > Thanks Fabian for starting the discussion.
> > >
> > > I would be biased towards option (1) that Stephan highlighted for the
> > > following reasons:
> > >
> > > - A separate github project is one more infrastructure to manage, and
> it
> > > lives outside the ASF. I would like to bring as much code as possible
> to
> > > the Apache Software Foundation, and not divide the codebase into two
> > > separate repositories.
> > >
> > > - The personal gratification (and thus motivation) is higher when
> > > contributing to a top-level Apache project than a github repository
> > > slightly associated with an ASF project. And contributors to the Flink
> > > project get karma that may lead to new committers, which is crucial as
> > the
> > > project is growing.
> > >
> > > Of course, non Apache-licensed contributions cannot be accepted. If we
> > have
> > > a good amount of those, we can start an infrastructure for Flink
> packages
> > > that lives outside the ASF, but I would wait for the need to come
> before
> > > doing this.
> > >
> > > My proposal would be to funnel contributions to the main repository
> (in a
> > > flink-contrib module) for now, including the recent contributions.
> > >
> > > Kostas
> > >
> > >
> > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
> wrote:
> > >
> > > > Yes, a "flink-contrib" project would be great.
> > > >
> > > > We have two options:
> > > >
> > > > 1) Make it part of the flink apache project.
> > > >   - PRO this makes it easy to get stuff for users
> > > >   - CONTRA this means stronger requirements on the code, blocker for
> > code
> > > > that uses dependencies under certain licenses, etc.
> > > >
> > > > 2) Make an independent github project.
> > > >  - PRO contributions can depended on more licenses, such as LGPL
> > > >  - PRO we can have more people that commit to this repo, committers
> can
> > > be
> > > > different from flink committers
> > > >  - CONTRA people need to grab the extensions from a different
> location
> > > >
> > > >
> > > > I am slightly biased towards (2), but open to both.
> > > >
> > > > Stephan
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <[hidden email]>
> > > > wrote:
> > > >
> > > > > I think top level maven module called "flink-contrib" is
> reasonable.
> > > > There
> > > > > are other projects having contrib package such as Akka, Django.
> > > > >
> > > > > Regards, Chiwan Park (Sent with iPhone)
> > > > >
> > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > we got a few contribution requests lately to add cool but
> > "non-core"
> > > > > > features to our API.
> > > > > > In previous discussions, concerns were raised to not bloat the
> APIs
> > > > with
> > > > > > too many "shortcut", "syntactic sugar", or special-case features.
> > > > > >
> > > > > > Instead we could setup a place to add Input/OutputFormats, common
> > > > > > operations, etc. which does not need as much control as the core
> > > APIs.
> > > > > Open
> > > > > > questions are:
> > > > > > - How do we organize it? (top-level maven module, modules in
> > > > flink-java,
> > > > > > flink-scala, java packages in the API modules, ...)
> > > > > > - How do we name it? flink-utils, flink-packages, ...
> > > > > >
> > > > > > Any opinions on this?
> > > > > >
> > > > > > Cheers, Fabian
> > > > >
> > > >
> > >
> >
>

[Kostas Tzoumas-2]

+1 for a package repository in the long run as per Ted's suggestion, and
moving forward with what Robert proposes

On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]>
wrote:

> I'm also in favor of option 1) with a "flink-contrib" maven module.
>
> I agree with Ted that we should certainly think about establishing a highly
> visible, easy to contribute and easy to use infrastructure for all kinds of
> contributions around the project.
> But I suspect that we need some time to come up with a good architecture
> and infrastructure for that. Maybe this also comes as an outside
> contribution to Flink?
>
> To have something immediately, we should start with a "flink-contrib"
> module.
>
> One thing that I would like to discuss first is a clear set of rules for
> contributions into that module.
> Code contributions to "flink-contrib" need:
> - to be tested on a cluster (not only by single-jvm tests)
> - to have test cases (because otherwise we can not guarantee that they
> build with our changes
> - to be of use for others
> - to have some documentation
>
> I would not deploy the flink-contrib package in the standard flink
> distribution. Users will have to add them as a maven dependency.
>
>
>
>
> On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]>
> wrote:
>
> > As the community of flink add-ons grows, a CPAN or maven-like mechanism
> > might be a nice option.  That would let people download and install
> > extensions very fluidly.
> >
> > The argument for making Apache contributions is definitely valid, but the
> > argument for the agility of fostering independent projects is that
> projects
> > can gain lots of popularity very quickly this way.  CPAN, CRAN, pip,
> maven
> > and RubyGems can be argued to be critical components of the popularity of
> > Perl, R, Python, Java/Scala and Ruby respectively.
> >
> >
> >
> >
> > On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > I am also more in favor of option 1).
> > >
> > > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
> > >
> > > > Thanks Fabian for starting the discussion.
> > > >
> > > > I would be biased towards option (1) that Stephan highlighted for the
> > > > following reasons:
> > > >
> > > > - A separate github project is one more infrastructure to manage, and
> > it
> > > > lives outside the ASF. I would like to bring as much code as possible
> > to
> > > > the Apache Software Foundation, and not divide the codebase into two
> > > > separate repositories.
> > > >
> > > > - The personal gratification (and thus motivation) is higher when
> > > > contributing to a top-level Apache project than a github repository
> > > > slightly associated with an ASF project. And contributors to the
> Flink
> > > > project get karma that may lead to new committers, which is crucial
> as
> > > the
> > > > project is growing.
> > > >
> > > > Of course, non Apache-licensed contributions cannot be accepted. If
> we
> > > have
> > > > a good amount of those, we can start an infrastructure for Flink
> > packages
> > > > that lives outside the ASF, but I would wait for the need to come
> > before
> > > > doing this.
> > > >
> > > > My proposal would be to funnel contributions to the main repository
> > (in a
> > > > flink-contrib module) for now, including the recent contributions.
> > > >
> > > > Kostas
> > > >
> > > >
> > > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
> > wrote:
> > > >
> > > > > Yes, a "flink-contrib" project would be great.
> > > > >
> > > > > We have two options:
> > > > >
> > > > > 1) Make it part of the flink apache project.
> > > > >   - PRO this makes it easy to get stuff for users
> > > > >   - CONTRA this means stronger requirements on the code, blocker
> for
> > > code
> > > > > that uses dependencies under certain licenses, etc.
> > > > >
> > > > > 2) Make an independent github project.
> > > > >  - PRO contributions can depended on more licenses, such as LGPL
> > > > >  - PRO we can have more people that commit to this repo, committers
> > can
> > > > be
> > > > > different from flink committers
> > > > >  - CONTRA people need to grab the extensions from a different
> > location
> > > > >
> > > > >
> > > > > I am slightly biased towards (2), but open to both.
> > > > >
> > > > > Stephan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > I think top level maven module called "flink-contrib" is
> > reasonable.
> > > > > There
> > > > > > are other projects having contrib package such as Akka, Django.
> > > > > >
> > > > > > Regards, Chiwan Park (Sent with iPhone)
> > > > > >
> > > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > we got a few contribution requests lately to add cool but
> > > "non-core"
> > > > > > > features to our API.
> > > > > > > In previous discussions, concerns were raised to not bloat the
> > APIs
> > > > > with
> > > > > > > too many "shortcut", "syntactic sugar", or special-case
> features.
> > > > > > >
> > > > > > > Instead we could setup a place to add Input/OutputFormats,
> common
> > > > > > > operations, etc. which does not need as much control as the
> core
> > > > APIs.
> > > > > > Open
> > > > > > > questions are:
> > > > > > > - How do we organize it? (top-level maven module, modules in
> > > > > flink-java,
> > > > > > > flink-scala, java packages in the API modules, ...)
> > > > > > > - How do we name it? flink-utils, flink-packages, ...
> > > > > > >
> > > > > > > Any opinions on this?
> > > > > > >
> > > > > > > Cheers, Fabian
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Aljoscha Krettek-2]

+1 to what robert said

On Sun, Jan 25, 2015 at 8:54 PM, Kostas Tzoumas <[hidden email]> wrote:

> +1 for a package repository in the long run as per Ted's suggestion, and
> moving forward with what Robert proposes
>
> On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]>
> wrote:
>
>> I'm also in favor of option 1) with a "flink-contrib" maven module.
>>
>> I agree with Ted that we should certainly think about establishing a highly
>> visible, easy to contribute and easy to use infrastructure for all kinds of
>> contributions around the project.
>> But I suspect that we need some time to come up with a good architecture
>> and infrastructure for that. Maybe this also comes as an outside
>> contribution to Flink?
>>
>> To have something immediately, we should start with a "flink-contrib"
>> module.
>>
>> One thing that I would like to discuss first is a clear set of rules for
>> contributions into that module.
>> Code contributions to "flink-contrib" need:
>> - to be tested on a cluster (not only by single-jvm tests)
>> - to have test cases (because otherwise we can not guarantee that they
>> build with our changes
>> - to be of use for others
>> - to have some documentation
>>
>> I would not deploy the flink-contrib package in the standard flink
>> distribution. Users will have to add them as a maven dependency.
>>
>>
>>
>>
>> On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]>
>> wrote:
>>
>> > As the community of flink add-ons grows, a CPAN or maven-like mechanism
>> > might be a nice option.  That would let people download and install
>> > extensions very fluidly.
>> >
>> > The argument for making Apache contributions is definitely valid, but the
>> > argument for the agility of fostering independent projects is that
>> projects
>> > can gain lots of popularity very quickly this way.  CPAN, CRAN, pip,
>> maven
>> > and RubyGems can be argued to be critical components of the popularity of
>> > Perl, R, Python, Java/Scala and Ruby respectively.
>> >
>> >
>> >
>> >
>> > On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]>
>> wrote:
>> >
>> > > I am also more in favor of option 1).
>> > >
>> > > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
>> > >
>> > > > Thanks Fabian for starting the discussion.
>> > > >
>> > > > I would be biased towards option (1) that Stephan highlighted for the
>> > > > following reasons:
>> > > >
>> > > > - A separate github project is one more infrastructure to manage, and
>> > it
>> > > > lives outside the ASF. I would like to bring as much code as possible
>> > to
>> > > > the Apache Software Foundation, and not divide the codebase into two
>> > > > separate repositories.
>> > > >
>> > > > - The personal gratification (and thus motivation) is higher when
>> > > > contributing to a top-level Apache project than a github repository
>> > > > slightly associated with an ASF project. And contributors to the
>> Flink
>> > > > project get karma that may lead to new committers, which is crucial
>> as
>> > > the
>> > > > project is growing.
>> > > >
>> > > > Of course, non Apache-licensed contributions cannot be accepted. If
>> we
>> > > have
>> > > > a good amount of those, we can start an infrastructure for Flink
>> > packages
>> > > > that lives outside the ASF, but I would wait for the need to come
>> > before
>> > > > doing this.
>> > > >
>> > > > My proposal would be to funnel contributions to the main repository
>> > (in a
>> > > > flink-contrib module) for now, including the recent contributions.
>> > > >
>> > > > Kostas
>> > > >
>> > > >
>> > > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
>> > wrote:
>> > > >
>> > > > > Yes, a "flink-contrib" project would be great.
>> > > > >
>> > > > > We have two options:
>> > > > >
>> > > > > 1) Make it part of the flink apache project.
>> > > > >   - PRO this makes it easy to get stuff for users
>> > > > >   - CONTRA this means stronger requirements on the code, blocker
>> for
>> > > code
>> > > > > that uses dependencies under certain licenses, etc.
>> > > > >
>> > > > > 2) Make an independent github project.
>> > > > >  - PRO contributions can depended on more licenses, such as LGPL
>> > > > >  - PRO we can have more people that commit to this repo, committers
>> > can
>> > > > be
>> > > > > different from flink committers
>> > > > >  - CONTRA people need to grab the extensions from a different
>> > location
>> > > > >
>> > > > >
>> > > > > I am slightly biased towards (2), but open to both.
>> > > > >
>> > > > > Stephan
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <
>> [hidden email]>
>> > > > > wrote:
>> > > > >
>> > > > > > I think top level maven module called "flink-contrib" is
>> > reasonable.
>> > > > > There
>> > > > > > are other projects having contrib package such as Akka, Django.
>> > > > > >
>> > > > > > Regards, Chiwan Park (Sent with iPhone)
>> > > > > >
>> > > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > we got a few contribution requests lately to add cool but
>> > > "non-core"
>> > > > > > > features to our API.
>> > > > > > > In previous discussions, concerns were raised to not bloat the
>> > APIs
>> > > > > with
>> > > > > > > too many "shortcut", "syntactic sugar", or special-case
>> features.
>> > > > > > >
>> > > > > > > Instead we could setup a place to add Input/OutputFormats,
>> common
>> > > > > > > operations, etc. which does not need as much control as the
>> core
>> > > > APIs.
>> > > > > > Open
>> > > > > > > questions are:
>> > > > > > > - How do we organize it? (top-level maven module, modules in
>> > > > > flink-java,
>> > > > > > > flink-scala, java packages in the API modules, ...)
>> > > > > > > - How do we name it? flink-utils, flink-packages, ...
>> > > > > > >
>> > > > > > > Any opinions on this?
>> > > > > > >
>> > > > > > > Cheers, Fabian
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

[Henry Saputra]

+1 for the idea.

We need to make sure PMC of Flink maintains knowledge of standard
Flink distribution, hence the "flink-contrib" should not be part of
the release.

- Henry

On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]> wrote:

> I'm also in favor of option 1) with a "flink-contrib" maven module.
>
> I agree with Ted that we should certainly think about establishing a highly
> visible, easy to contribute and easy to use infrastructure for all kinds of
> contributions around the project.
> But I suspect that we need some time to come up with a good architecture
> and infrastructure for that. Maybe this also comes as an outside
> contribution to Flink?
>
> To have something immediately, we should start with a "flink-contrib"
> module.
>
> One thing that I would like to discuss first is a clear set of rules for
> contributions into that module.
> Code contributions to "flink-contrib" need:
> - to be tested on a cluster (not only by single-jvm tests)
> - to have test cases (because otherwise we can not guarantee that they
> build with our changes
> - to be of use for others
> - to have some documentation
>
> I would not deploy the flink-contrib package in the standard flink
> distribution. Users will have to add them as a maven dependency.
>
>
>
>
> On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]> wrote:
>
>> As the community of flink add-ons grows, a CPAN or maven-like mechanism
>> might be a nice option.  That would let people download and install
>> extensions very fluidly.
>>
>> The argument for making Apache contributions is definitely valid, but the
>> argument for the agility of fostering independent projects is that projects
>> can gain lots of popularity very quickly this way.  CPAN, CRAN, pip, maven
>> and RubyGems can be argued to be critical components of the popularity of
>> Perl, R, Python, Java/Scala and Ruby respectively.
>>
>>
>>
>>
>> On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]> wrote:
>>
>> > I am also more in favor of option 1).
>> >
>> > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
>> >
>> > > Thanks Fabian for starting the discussion.
>> > >
>> > > I would be biased towards option (1) that Stephan highlighted for the
>> > > following reasons:
>> > >
>> > > - A separate github project is one more infrastructure to manage, and
>> it
>> > > lives outside the ASF. I would like to bring as much code as possible
>> to
>> > > the Apache Software Foundation, and not divide the codebase into two
>> > > separate repositories.
>> > >
>> > > - The personal gratification (and thus motivation) is higher when
>> > > contributing to a top-level Apache project than a github repository
>> > > slightly associated with an ASF project. And contributors to the Flink
>> > > project get karma that may lead to new committers, which is crucial as
>> > the
>> > > project is growing.
>> > >
>> > > Of course, non Apache-licensed contributions cannot be accepted. If we
>> > have
>> > > a good amount of those, we can start an infrastructure for Flink
>> packages
>> > > that lives outside the ASF, but I would wait for the need to come
>> before
>> > > doing this.
>> > >
>> > > My proposal would be to funnel contributions to the main repository
>> (in a
>> > > flink-contrib module) for now, including the recent contributions.
>> > >
>> > > Kostas
>> > >
>> > >
>> > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
>> wrote:
>> > >
>> > > > Yes, a "flink-contrib" project would be great.
>> > > >
>> > > > We have two options:
>> > > >
>> > > > 1) Make it part of the flink apache project.
>> > > >   - PRO this makes it easy to get stuff for users
>> > > >   - CONTRA this means stronger requirements on the code, blocker for
>> > code
>> > > > that uses dependencies under certain licenses, etc.
>> > > >
>> > > > 2) Make an independent github project.
>> > > >  - PRO contributions can depended on more licenses, such as LGPL
>> > > >  - PRO we can have more people that commit to this repo, committers
>> can
>> > > be
>> > > > different from flink committers
>> > > >  - CONTRA people need to grab the extensions from a different
>> location
>> > > >
>> > > >
>> > > > I am slightly biased towards (2), but open to both.
>> > > >
>> > > > Stephan
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <[hidden email]>
>> > > > wrote:
>> > > >
>> > > > > I think top level maven module called "flink-contrib" is
>> reasonable.
>> > > > There
>> > > > > are other projects having contrib package such as Akka, Django.
>> > > > >
>> > > > > Regards, Chiwan Park (Sent with iPhone)
>> > > > >
>> > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > we got a few contribution requests lately to add cool but
>> > "non-core"
>> > > > > > features to our API.
>> > > > > > In previous discussions, concerns were raised to not bloat the
>> APIs
>> > > > with
>> > > > > > too many "shortcut", "syntactic sugar", or special-case features.
>> > > > > >
>> > > > > > Instead we could setup a place to add Input/OutputFormats, common
>> > > > > > operations, etc. which does not need as much control as the core
>> > > APIs.
>> > > > > Open
>> > > > > > questions are:
>> > > > > > - How do we organize it? (top-level maven module, modules in
>> > > > flink-java,
>> > > > > > flink-scala, java packages in the API modules, ...)
>> > > > > > - How do we name it? flink-utils, flink-packages, ...
>> > > > > >
>> > > > > > Any opinions on this?
>> > > > > >
>> > > > > > Cheers, Fabian
>> > > > >
>> > > >
>> > >
>> >
>>

[Till Rohrmann]

+1 for Robert's proposal.

On Mon, Jan 26, 2015 at 5:43 AM, Henry Saputra <[hidden email]>
wrote:

> +1 for the idea.
>
> We need to make sure PMC of Flink maintains knowledge of standard
> Flink distribution, hence the "flink-contrib" should not be part of
> the release.
>
> - Henry
>
> On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]>
> wrote:
> > I'm also in favor of option 1) with a "flink-contrib" maven module.
> >
> > I agree with Ted that we should certainly think about establishing a
> highly
> > visible, easy to contribute and easy to use infrastructure for all kinds
> of
> > contributions around the project.
> > But I suspect that we need some time to come up with a good architecture
> > and infrastructure for that. Maybe this also comes as an outside
> > contribution to Flink?
> >
> > To have something immediately, we should start with a "flink-contrib"
> > module.
> >
> > One thing that I would like to discuss first is a clear set of rules for
> > contributions into that module.
> > Code contributions to "flink-contrib" need:
> > - to be tested on a cluster (not only by single-jvm tests)
> > - to have test cases (because otherwise we can not guarantee that they
> > build with our changes
> > - to be of use for others
> > - to have some documentation
> >
> > I would not deploy the flink-contrib package in the standard flink
> > distribution. Users will have to add them as a maven dependency.
> >
> >
> >
> >
> > On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]>
> wrote:
> >
> >> As the community of flink add-ons grows, a CPAN or maven-like mechanism
> >> might be a nice option.  That would let people download and install
> >> extensions very fluidly.
> >>
> >> The argument for making Apache contributions is definitely valid, but
> the
> >> argument for the agility of fostering independent projects is that
> projects
> >> can gain lots of popularity very quickly this way.  CPAN, CRAN, pip,
> maven
> >> and RubyGems can be argued to be critical components of the popularity
> of
> >> Perl, R, Python, Java/Scala and Ruby respectively.
> >>
> >>
> >>
> >>
> >> On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]>
> wrote:
> >>
> >> > I am also more in favor of option 1).
> >> >
> >> > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
> >> >
> >> > > Thanks Fabian for starting the discussion.
> >> > >
> >> > > I would be biased towards option (1) that Stephan highlighted for
> the
> >> > > following reasons:
> >> > >
> >> > > - A separate github project is one more infrastructure to manage,
> and
> >> it
> >> > > lives outside the ASF. I would like to bring as much code as
> possible
> >> to
> >> > > the Apache Software Foundation, and not divide the codebase into two
> >> > > separate repositories.
> >> > >
> >> > > - The personal gratification (and thus motivation) is higher when
> >> > > contributing to a top-level Apache project than a github repository
> >> > > slightly associated with an ASF project. And contributors to the
> Flink
> >> > > project get karma that may lead to new committers, which is crucial
> as
> >> > the
> >> > > project is growing.
> >> > >
> >> > > Of course, non Apache-licensed contributions cannot be accepted. If
> we
> >> > have
> >> > > a good amount of those, we can start an infrastructure for Flink
> >> packages
> >> > > that lives outside the ASF, but I would wait for the need to come
> >> before
> >> > > doing this.
> >> > >
> >> > > My proposal would be to funnel contributions to the main repository
> >> (in a
> >> > > flink-contrib module) for now, including the recent contributions.
> >> > >
> >> > > Kostas
> >> > >
> >> > >
> >> > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
> >> wrote:
> >> > >
> >> > > > Yes, a "flink-contrib" project would be great.
> >> > > >
> >> > > > We have two options:
> >> > > >
> >> > > > 1) Make it part of the flink apache project.
> >> > > >   - PRO this makes it easy to get stuff for users
> >> > > >   - CONTRA this means stronger requirements on the code, blocker
> for
> >> > code
> >> > > > that uses dependencies under certain licenses, etc.
> >> > > >
> >> > > > 2) Make an independent github project.
> >> > > >  - PRO contributions can depended on more licenses, such as LGPL
> >> > > >  - PRO we can have more people that commit to this repo,
> committers
> >> can
> >> > > be
> >> > > > different from flink committers
> >> > > >  - CONTRA people need to grab the extensions from a different
> >> location
> >> > > >
> >> > > >
> >> > > > I am slightly biased towards (2), but open to both.
> >> > > >
> >> > > > Stephan
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <
> [hidden email]>
> >> > > > wrote:
> >> > > >
> >> > > > > I think top level maven module called "flink-contrib" is
> >> reasonable.
> >> > > > There
> >> > > > > are other projects having contrib package such as Akka, Django.
> >> > > > >
> >> > > > > Regards, Chiwan Park (Sent with iPhone)
> >> > > > >
> >> > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> >> > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > we got a few contribution requests lately to add cool but
> >> > "non-core"
> >> > > > > > features to our API.
> >> > > > > > In previous discussions, concerns were raised to not bloat the
> >> APIs
> >> > > > with
> >> > > > > > too many "shortcut", "syntactic sugar", or special-case
> features.
> >> > > > > >
> >> > > > > > Instead we could setup a place to add Input/OutputFormats,
> common
> >> > > > > > operations, etc. which does not need as much control as the
> core
> >> > > APIs.
> >> > > > > Open
> >> > > > > > questions are:
> >> > > > > > - How do we organize it? (top-level maven module, modules in
> >> > > > flink-java,
> >> > > > > > flink-scala, java packages in the API modules, ...)
> >> > > > > > - How do we name it? flink-utils, flink-packages, ...
> >> > > > > >
> >> > > > > > Any opinions on this?
> >> > > > > >
> >> > > > > > Cheers, Fabian
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>

[Robert Metzger]

I've added a JIRA issue to create the module:
https://issues.apache.org/jira/browse/FLINK-1452


On Mon, Jan 26, 2015 at 11:39 AM, Till Rohrmann <[hidden email]>
wrote:

> +1 for Robert's proposal.
>
> On Mon, Jan 26, 2015 at 5:43 AM, Henry Saputra <[hidden email]>
> wrote:
>
> > +1 for the idea.
> >
> > We need to make sure PMC of Flink maintains knowledge of standard
> > Flink distribution, hence the "flink-contrib" should not be part of
> > the release.
> >
> > - Henry
> >
> > On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]>
> > wrote:
> > > I'm also in favor of option 1) with a "flink-contrib" maven module.
> > >
> > > I agree with Ted that we should certainly think about establishing a
> > highly
> > > visible, easy to contribute and easy to use infrastructure for all
> kinds
> > of
> > > contributions around the project.
> > > But I suspect that we need some time to come up with a good
> architecture
> > > and infrastructure for that. Maybe this also comes as an outside
> > > contribution to Flink?
> > >
> > > To have something immediately, we should start with a "flink-contrib"
> > > module.
> > >
> > > One thing that I would like to discuss first is a clear set of rules
> for
> > > contributions into that module.
> > > Code contributions to "flink-contrib" need:
> > > - to be tested on a cluster (not only by single-jvm tests)
> > > - to have test cases (because otherwise we can not guarantee that they
> > > build with our changes
> > > - to be of use for others
> > > - to have some documentation
> > >
> > > I would not deploy the flink-contrib package in the standard flink
> > > distribution. Users will have to add them as a maven dependency.
> > >
> > >
> > >
> > >
> > > On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]>
> > wrote:
> > >
> > >> As the community of flink add-ons grows, a CPAN or maven-like
> mechanism
> > >> might be a nice option.  That would let people download and install
> > >> extensions very fluidly.
> > >>
> > >> The argument for making Apache contributions is definitely valid, but
> > the
> > >> argument for the agility of fostering independent projects is that
> > projects
> > >> can gain lots of popularity very quickly this way.  CPAN, CRAN, pip,
> > maven
> > >> and RubyGems can be argued to be critical components of the popularity
> > of
> > >> Perl, R, Python, Java/Scala and Ruby respectively.
> > >>
> > >>
> > >>
> > >>
> > >> On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]>
> > wrote:
> > >>
> > >> > I am also more in favor of option 1).
> > >> >
> > >> > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
> > >> >
> > >> > > Thanks Fabian for starting the discussion.
> > >> > >
> > >> > > I would be biased towards option (1) that Stephan highlighted for
> > the
> > >> > > following reasons:
> > >> > >
> > >> > > - A separate github project is one more infrastructure to manage,
> > and
> > >> it
> > >> > > lives outside the ASF. I would like to bring as much code as
> > possible
> > >> to
> > >> > > the Apache Software Foundation, and not divide the codebase into
> two
> > >> > > separate repositories.
> > >> > >
> > >> > > - The personal gratification (and thus motivation) is higher when
> > >> > > contributing to a top-level Apache project than a github
> repository
> > >> > > slightly associated with an ASF project. And contributors to the
> > Flink
> > >> > > project get karma that may lead to new committers, which is
> crucial
> > as
> > >> > the
> > >> > > project is growing.
> > >> > >
> > >> > > Of course, non Apache-licensed contributions cannot be accepted.
> If
> > we
> > >> > have
> > >> > > a good amount of those, we can start an infrastructure for Flink
> > >> packages
> > >> > > that lives outside the ASF, but I would wait for the need to come
> > >> before
> > >> > > doing this.
> > >> > >
> > >> > > My proposal would be to funnel contributions to the main
> repository
> > >> (in a
> > >> > > flink-contrib module) for now, including the recent contributions.
> > >> > >
> > >> > > Kostas
> > >> > >
> > >> > >
> > >> > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
> > >> wrote:
> > >> > >
> > >> > > > Yes, a "flink-contrib" project would be great.
> > >> > > >
> > >> > > > We have two options:
> > >> > > >
> > >> > > > 1) Make it part of the flink apache project.
> > >> > > >   - PRO this makes it easy to get stuff for users
> > >> > > >   - CONTRA this means stronger requirements on the code, blocker
> > for
> > >> > code
> > >> > > > that uses dependencies under certain licenses, etc.
> > >> > > >
> > >> > > > 2) Make an independent github project.
> > >> > > >  - PRO contributions can depended on more licenses, such as LGPL
> > >> > > >  - PRO we can have more people that commit to this repo,
> > committers
> > >> can
> > >> > > be
> > >> > > > different from flink committers
> > >> > > >  - CONTRA people need to grab the extensions from a different
> > >> location
> > >> > > >
> > >> > > >
> > >> > > > I am slightly biased towards (2), but open to both.
> > >> > > >
> > >> > > > Stephan
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <
> > [hidden email]>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > I think top level maven module called "flink-contrib" is
> > >> reasonable.
> > >> > > > There
> > >> > > > > are other projects having contrib package such as Akka,
> Django.
> > >> > > > >
> > >> > > > > Regards, Chiwan Park (Sent with iPhone)
> > >> > > > >
> > >> > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> > >> > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > we got a few contribution requests lately to add cool but
> > >> > "non-core"
> > >> > > > > > features to our API.
> > >> > > > > > In previous discussions, concerns were raised to not bloat
> the
> > >> APIs
> > >> > > > with
> > >> > > > > > too many "shortcut", "syntactic sugar", or special-case
> > features.
> > >> > > > > >
> > >> > > > > > Instead we could setup a place to add Input/OutputFormats,
> > common
> > >> > > > > > operations, etc. which does not need as much control as the
> > core
> > >> > > APIs.
> > >> > > > > Open
> > >> > > > > > questions are:
> > >> > > > > > - How do we organize it? (top-level maven module, modules in
> > >> > > > flink-java,
> > >> > > > > > flink-scala, java packages in the API modules, ...)
> > >> > > > > > - How do we name it? flink-utils, flink-packages, ...
> > >> > > > > >
> > >> > > > > > Any opinions on this?
> > >> > > > > >
> > >> > > > > > Cheers, Fabian
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>

[Max Michels]

+1 for having an optional flink-contrib maven dependency and an
extension repository in the long run.

On Mon, Jan 26, 2015 at 12:00 PM, Robert Metzger <[hidden email]> wrote:

> I've added a JIRA issue to create the module:
> https://issues.apache.org/jira/browse/FLINK-1452
>
>
> On Mon, Jan 26, 2015 at 11:39 AM, Till Rohrmann <[hidden email]>
> wrote:
>
>> +1 for Robert's proposal.
>>
>> On Mon, Jan 26, 2015 at 5:43 AM, Henry Saputra <[hidden email]>
>> wrote:
>>
>> > +1 for the idea.
>> >
>> > We need to make sure PMC of Flink maintains knowledge of standard
>> > Flink distribution, hence the "flink-contrib" should not be part of
>> > the release.
>> >
>> > - Henry
>> >
>> > On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]>
>> > wrote:
>> > > I'm also in favor of option 1) with a "flink-contrib" maven module.
>> > >
>> > > I agree with Ted that we should certainly think about establishing a
>> > highly
>> > > visible, easy to contribute and easy to use infrastructure for all
>> kinds
>> > of
>> > > contributions around the project.
>> > > But I suspect that we need some time to come up with a good
>> architecture
>> > > and infrastructure for that. Maybe this also comes as an outside
>> > > contribution to Flink?
>> > >
>> > > To have something immediately, we should start with a "flink-contrib"
>> > > module.
>> > >
>> > > One thing that I would like to discuss first is a clear set of rules
>> for
>> > > contributions into that module.
>> > > Code contributions to "flink-contrib" need:
>> > > - to be tested on a cluster (not only by single-jvm tests)
>> > > - to have test cases (because otherwise we can not guarantee that they
>> > > build with our changes
>> > > - to be of use for others
>> > > - to have some documentation
>> > >
>> > > I would not deploy the flink-contrib package in the standard flink
>> > > distribution. Users will have to add them as a maven dependency.
>> > >
>> > >
>> > >
>> > >
>> > > On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <[hidden email]>
>> > wrote:
>> > >
>> > >> As the community of flink add-ons grows, a CPAN or maven-like
>> mechanism
>> > >> might be a nice option.  That would let people download and install
>> > >> extensions very fluidly.
>> > >>
>> > >> The argument for making Apache contributions is definitely valid, but
>> > the
>> > >> argument for the agility of fostering independent projects is that
>> > projects
>> > >> can gain lots of popularity very quickly this way.  CPAN, CRAN, pip,
>> > maven
>> > >> and RubyGems can be argued to be critical components of the popularity
>> > of
>> > >> Perl, R, Python, Java/Scala and Ruby respectively.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]>
>> > wrote:
>> > >>
>> > >> > I am also more in favor of option 1).
>> > >> >
>> > >> > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
>> > >> >
>> > >> > > Thanks Fabian for starting the discussion.
>> > >> > >
>> > >> > > I would be biased towards option (1) that Stephan highlighted for
>> > the
>> > >> > > following reasons:
>> > >> > >
>> > >> > > - A separate github project is one more infrastructure to manage,
>> > and
>> > >> it
>> > >> > > lives outside the ASF. I would like to bring as much code as
>> > possible
>> > >> to
>> > >> > > the Apache Software Foundation, and not divide the codebase into
>> two
>> > >> > > separate repositories.
>> > >> > >
>> > >> > > - The personal gratification (and thus motivation) is higher when
>> > >> > > contributing to a top-level Apache project than a github
>> repository
>> > >> > > slightly associated with an ASF project. And contributors to the
>> > Flink
>> > >> > > project get karma that may lead to new committers, which is
>> crucial
>> > as
>> > >> > the
>> > >> > > project is growing.
>> > >> > >
>> > >> > > Of course, non Apache-licensed contributions cannot be accepted.
>> If
>> > we
>> > >> > have
>> > >> > > a good amount of those, we can start an infrastructure for Flink
>> > >> packages
>> > >> > > that lives outside the ASF, but I would wait for the need to come
>> > >> before
>> > >> > > doing this.
>> > >> > >
>> > >> > > My proposal would be to funnel contributions to the main
>> repository
>> > >> (in a
>> > >> > > flink-contrib module) for now, including the recent contributions.
>> > >> > >
>> > >> > > Kostas
>> > >> > >
>> > >> > >
>> > >> > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <[hidden email]>
>> > >> wrote:
>> > >> > >
>> > >> > > > Yes, a "flink-contrib" project would be great.
>> > >> > > >
>> > >> > > > We have two options:
>> > >> > > >
>> > >> > > > 1) Make it part of the flink apache project.
>> > >> > > >   - PRO this makes it easy to get stuff for users
>> > >> > > >   - CONTRA this means stronger requirements on the code, blocker
>> > for
>> > >> > code
>> > >> > > > that uses dependencies under certain licenses, etc.
>> > >> > > >
>> > >> > > > 2) Make an independent github project.
>> > >> > > >  - PRO contributions can depended on more licenses, such as LGPL
>> > >> > > >  - PRO we can have more people that commit to this repo,
>> > committers
>> > >> can
>> > >> > > be
>> > >> > > > different from flink committers
>> > >> > > >  - CONTRA people need to grab the extensions from a different
>> > >> location
>> > >> > > >
>> > >> > > >
>> > >> > > > I am slightly biased towards (2), but open to both.
>> > >> > > >
>> > >> > > > Stephan
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <
>> > [hidden email]>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > I think top level maven module called "flink-contrib" is
>> > >> reasonable.
>> > >> > > > There
>> > >> > > > > are other projects having contrib package such as Akka,
>> Django.
>> > >> > > > >
>> > >> > > > > Regards, Chiwan Park (Sent with iPhone)
>> > >> > > > >
>> > >> > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
>> > >> > > > >
>> > >> > > > > > Hi all,
>> > >> > > > > >
>> > >> > > > > > we got a few contribution requests lately to add cool but
>> > >> > "non-core"
>> > >> > > > > > features to our API.
>> > >> > > > > > In previous discussions, concerns were raised to not bloat
>> the
>> > >> APIs
>> > >> > > > with
>> > >> > > > > > too many "shortcut", "syntactic sugar", or special-case
>> > features.
>> > >> > > > > >
>> > >> > > > > > Instead we could setup a place to add Input/OutputFormats,
>> > common
>> > >> > > > > > operations, etc. which does not need as much control as the
>> > core
>> > >> > > APIs.
>> > >> > > > > Open
>> > >> > > > > > questions are:
>> > >> > > > > > - How do we organize it? (top-level maven module, modules in
>> > >> > > > flink-java,
>> > >> > > > > > flink-scala, java packages in the API modules, ...)
>> > >> > > > > > - How do we name it? flink-utils, flink-packages, ...
>> > >> > > > > >
>> > >> > > > > > Any opinions on this?
>> > >> > > > > >
>> > >> > > > > > Cheers, Fabian
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>

[Robert Metzger]

Hey guys,

I think the JIRA associated to this discussion needs some attention:
https://issues.apache.org/jira/browse/FLINK-1452


On Mon, Jan 26, 2015 at 2:24 PM, Max Michels <[hidden email]> wrote:

> +1 for having an optional flink-contrib maven dependency and an
> extension repository in the long run.
>
> On Mon, Jan 26, 2015 at 12:00 PM, Robert Metzger <[hidden email]>
> wrote:
> > I've added a JIRA issue to create the module:
> > https://issues.apache.org/jira/browse/FLINK-1452
> >
> >
> > On Mon, Jan 26, 2015 at 11:39 AM, Till Rohrmann <[hidden email]>
> > wrote:
> >
> >> +1 for Robert's proposal.
> >>
> >> On Mon, Jan 26, 2015 at 5:43 AM, Henry Saputra <[hidden email]
> >
> >> wrote:
> >>
> >> > +1 for the idea.
> >> >
> >> > We need to make sure PMC of Flink maintains knowledge of standard
> >> > Flink distribution, hence the "flink-contrib" should not be part of
> >> > the release.
> >> >
> >> > - Henry
> >> >
> >> > On Sun, Jan 25, 2015 at 10:33 AM, Robert Metzger <[hidden email]
> >
> >> > wrote:
> >> > > I'm also in favor of option 1) with a "flink-contrib" maven module.
> >> > >
> >> > > I agree with Ted that we should certainly think about establishing a
> >> > highly
> >> > > visible, easy to contribute and easy to use infrastructure for all
> >> kinds
> >> > of
> >> > > contributions around the project.
> >> > > But I suspect that we need some time to come up with a good
> >> architecture
> >> > > and infrastructure for that. Maybe this also comes as an outside
> >> > > contribution to Flink?
> >> > >
> >> > > To have something immediately, we should start with a
> "flink-contrib"
> >> > > module.
> >> > >
> >> > > One thing that I would like to discuss first is a clear set of rules
> >> for
> >> > > contributions into that module.
> >> > > Code contributions to "flink-contrib" need:
> >> > > - to be tested on a cluster (not only by single-jvm tests)
> >> > > - to have test cases (because otherwise we can not guarantee that
> they
> >> > > build with our changes
> >> > > - to be of use for others
> >> > > - to have some documentation
> >> > >
> >> > > I would not deploy the flink-contrib package in the standard flink
> >> > > distribution. Users will have to add them as a maven dependency.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Sat, Jan 24, 2015 at 11:02 PM, Ted Dunning <
> [hidden email]>
> >> > wrote:
> >> > >
> >> > >> As the community of flink add-ons grows, a CPAN or maven-like
> >> mechanism
> >> > >> might be a nice option.  That would let people download and install
> >> > >> extensions very fluidly.
> >> > >>
> >> > >> The argument for making Apache contributions is definitely valid,
> but
> >> > the
> >> > >> argument for the agility of fostering independent projects is that
> >> > projects
> >> > >> can gain lots of popularity very quickly this way.  CPAN, CRAN,
> pip,
> >> > maven
> >> > >> and RubyGems can be argued to be critical components of the
> popularity
> >> > of
> >> > >> Perl, R, Python, Java/Scala and Ruby respectively.
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Sat, Jan 24, 2015 at 4:26 PM, Fabian Hueske <[hidden email]>
> >> > wrote:
> >> > >>
> >> > >> > I am also more in favor of option 1).
> >> > >> >
> >> > >> > 2015-01-24 20:27 GMT+01:00 Kostas Tzoumas <[hidden email]>:
> >> > >> >
> >> > >> > > Thanks Fabian for starting the discussion.
> >> > >> > >
> >> > >> > > I would be biased towards option (1) that Stephan highlighted
> for
> >> > the
> >> > >> > > following reasons:
> >> > >> > >
> >> > >> > > - A separate github project is one more infrastructure to
> manage,
> >> > and
> >> > >> it
> >> > >> > > lives outside the ASF. I would like to bring as much code as
> >> > possible
> >> > >> to
> >> > >> > > the Apache Software Foundation, and not divide the codebase
> into
> >> two
> >> > >> > > separate repositories.
> >> > >> > >
> >> > >> > > - The personal gratification (and thus motivation) is higher
> when
> >> > >> > > contributing to a top-level Apache project than a github
> >> repository
> >> > >> > > slightly associated with an ASF project. And contributors to
> the
> >> > Flink
> >> > >> > > project get karma that may lead to new committers, which is
> >> crucial
> >> > as
> >> > >> > the
> >> > >> > > project is growing.
> >> > >> > >
> >> > >> > > Of course, non Apache-licensed contributions cannot be
> accepted.
> >> If
> >> > we
> >> > >> > have
> >> > >> > > a good amount of those, we can start an infrastructure for
> Flink
> >> > >> packages
> >> > >> > > that lives outside the ASF, but I would wait for the need to
> come
> >> > >> before
> >> > >> > > doing this.
> >> > >> > >
> >> > >> > > My proposal would be to funnel contributions to the main
> >> repository
> >> > >> (in a
> >> > >> > > flink-contrib module) for now, including the recent
> contributions.
> >> > >> > >
> >> > >> > > Kostas
> >> > >> > >
> >> > >> > >
> >> > >> > > On Sat, Jan 24, 2015 at 11:15 AM, Stephan Ewen <
> [hidden email]>
> >> > >> wrote:
> >> > >> > >
> >> > >> > > > Yes, a "flink-contrib" project would be great.
> >> > >> > > >
> >> > >> > > > We have two options:
> >> > >> > > >
> >> > >> > > > 1) Make it part of the flink apache project.
> >> > >> > > >   - PRO this makes it easy to get stuff for users
> >> > >> > > >   - CONTRA this means stronger requirements on the code,
> blocker
> >> > for
> >> > >> > code
> >> > >> > > > that uses dependencies under certain licenses, etc.
> >> > >> > > >
> >> > >> > > > 2) Make an independent github project.
> >> > >> > > >  - PRO contributions can depended on more licenses, such as
> LGPL
> >> > >> > > >  - PRO we can have more people that commit to this repo,
> >> > committers
> >> > >> can
> >> > >> > > be
> >> > >> > > > different from flink committers
> >> > >> > > >  - CONTRA people need to grab the extensions from a different
> >> > >> location
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > I am slightly biased towards (2), but open to both.
> >> > >> > > >
> >> > >> > > > Stephan
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Sat, Jan 24, 2015 at 5:29 AM, Chiwan Park <
> >> > [hidden email]>
> >> > >> > > > wrote:
> >> > >> > > >
> >> > >> > > > > I think top level maven module called "flink-contrib" is
> >> > >> reasonable.
> >> > >> > > > There
> >> > >> > > > > are other projects having contrib package such as Akka,
> >> Django.
> >> > >> > > > >
> >> > >> > > > > Regards, Chiwan Park (Sent with iPhone)
> >> > >> > > > >
> >> > >> > > > > 2015. 1. 24. 오후 7:15 Fabian Hueske <[hidden email]> 작성:
> >> > >> > > > >
> >> > >> > > > > > Hi all,
> >> > >> > > > > >
> >> > >> > > > > > we got a few contribution requests lately to add cool but
> >> > >> > "non-core"
> >> > >> > > > > > features to our API.
> >> > >> > > > > > In previous discussions, concerns were raised to not
> bloat
> >> the
> >> > >> APIs
> >> > >> > > > with
> >> > >> > > > > > too many "shortcut", "syntactic sugar", or special-case
> >> > features.
> >> > >> > > > > >
> >> > >> > > > > > Instead we could setup a place to add
> Input/OutputFormats,
> >> > common
> >> > >> > > > > > operations, etc. which does not need as much control as
> the
> >> > core
> >> > >> > > APIs.
> >> > >> > > > > Open
> >> > >> > > > > > questions are:
> >> > >> > > > > > - How do we organize it? (top-level maven module,
> modules in
> >> > >> > > > flink-java,
> >> > >> > > > > > flink-scala, java packages in the API modules, ...)
> >> > >> > > > > > - How do we name it? flink-utils, flink-packages, ...
> >> > >> > > > > >
> >> > >> > > > > > Any opinions on this?
> >> > >> > > > > >
> >> > >> > > > > > Cheers, Fabian
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>