[DISCUSS] flink-external

[Matthias J. Sax-2]

Hi,

many people are building quite exiting stuff on top of Flink. It is hard
to keep an good overview on what stuff is available and what not. What
do you think about starting a second git repository "flink-external"
that collects all those code?

The ideas would be to collect stuff in a central point, such that people
can access it easily and get an overview what is already available (this
might also avoid duplicate development). It might also be a good point
to show common patterns. In order to collect as much as possible, the
contributing requirement (with respect to testing etc) could be lower
than for Flink itself.

For example, I recently started a small flink-clojure module with a
simple word-count example to answer a question on SO. Including this in
Flink would not be appropriate. However, for a flink-external repro it
might be nice to have.

What do you think about it?


-Matthias

[mxm]

Hi Matthias,

Thanks for bringing up this idea. Actually, it has been discussed a
couple of times on the mailing list whether we should have a central
place for third-party extensions/contributions/libraries. This could
either be something package-based or, like you proposed, another
repository.

An external place for contributions raises a couple of questions

- Which version should the external contributions be based on?
- How do we make sure, the extensions are continuously updated?
(dedicated maintainers or automatic compatibility checks)
- How do we easily plug-in the external modules into Flink?

In the long term, we really need a solution for these questions. The
code base of Flink is growing and more and more packages go to
flink-contrib/flink-staging. I would find something packaged-based
better than a repository. Quite frankly, momentarily, I think
developing such a plugin system is out of scope for most Flink
developers. At the current pace of Flink development, collecting these
contributions externally without properly maintaining them, doesn't
make much sense to me.

Cheers,
Max



On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:

>
> Hi,
>
> many people are building quite exiting stuff on top of Flink. It is hard
> to keep an good overview on what stuff is available and what not. What
> do you think about starting a second git repository "flink-external"
> that collects all those code?
>
> The ideas would be to collect stuff in a central point, such that people
> can access it easily and get an overview what is already available (this
> might also avoid duplicate development). It might also be a good point
> to show common patterns. In order to collect as much as possible, the
> contributing requirement (with respect to testing etc) could be lower
> than for Flink itself.
>
> For example, I recently started a small flink-clojure module with a
> simple word-count example to answer a question on SO. Including this in
> Flink would not be appropriate. However, for a flink-external repro it
> might be nice to have.
>
> What do you think about it?
>
>
> -Matthias
>

[Vasiliki Kalavri]

How about, for now, we simply create a page where we gather links/short
descriptions of all these contributions
and let the maintenance and dependency management to the tool/library
creators?
This way we will at least have these contributions in one place and link to
them somewhere from the website.

-Vasia.

On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:

> Hi Matthias,
>
> Thanks for bringing up this idea. Actually, it has been discussed a
> couple of times on the mailing list whether we should have a central
> place for third-party extensions/contributions/libraries. This could
> either be something package-based or, like you proposed, another
> repository.
>
> An external place for contributions raises a couple of questions
>
> - Which version should the external contributions be based on?
> - How do we make sure, the extensions are continuously updated?
> (dedicated maintainers or automatic compatibility checks)
> - How do we easily plug-in the external modules into Flink?
>
> In the long term, we really need a solution for these questions. The
> code base of Flink is growing and more and more packages go to
> flink-contrib/flink-staging. I would find something packaged-based
> better than a repository. Quite frankly, momentarily, I think
> developing such a plugin system is out of scope for most Flink
> developers. At the current pace of Flink development, collecting these
> contributions externally without properly maintaining them, doesn't
> make much sense to me.
>
> Cheers,
> Max
>
>
>
> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
> >
> > Hi,
> >
> > many people are building quite exiting stuff on top of Flink. It is hard
> > to keep an good overview on what stuff is available and what not. What
> > do you think about starting a second git repository "flink-external"
> > that collects all those code?
> >
> > The ideas would be to collect stuff in a central point, such that people
> > can access it easily and get an overview what is already available (this
> > might also avoid duplicate development). It might also be a good point
> > to show common patterns. In order to collect as much as possible, the
> > contributing requirement (with respect to testing etc) could be lower
> > than for Flink itself.
> >
> > For example, I recently started a small flink-clojure module with a
> > simple word-count example to answer a question on SO. Including this in
> > Flink would not be appropriate. However, for a flink-external repro it
> > might be nice to have.
> >
> > What do you think about it?
> >
> >
> > -Matthias
> >
>

[Fabian Hueske-2]

+1 for Vasia's suggestion. It's a good and pragmatic start.

Putting such contributions into an Apache-managed repository might have
some side effects.

2015-10-08 12:15 GMT+02:00 Vasiliki Kalavri <[hidden email]>:

> How about, for now, we simply create a page where we gather links/short
> descriptions of all these contributions
> and let the maintenance and dependency management to the tool/library
> creators?
> This way we will at least have these contributions in one place and link to
> them somewhere from the website.
>
> -Vasia.
>
> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>
> > Hi Matthias,
> >
> > Thanks for bringing up this idea. Actually, it has been discussed a
> > couple of times on the mailing list whether we should have a central
> > place for third-party extensions/contributions/libraries. This could
> > either be something package-based or, like you proposed, another
> > repository.
> >
> > An external place for contributions raises a couple of questions
> >
> > - Which version should the external contributions be based on?
> > - How do we make sure, the extensions are continuously updated?
> > (dedicated maintainers or automatic compatibility checks)
> > - How do we easily plug-in the external modules into Flink?
> >
> > In the long term, we really need a solution for these questions. The
> > code base of Flink is growing and more and more packages go to
> > flink-contrib/flink-staging. I would find something packaged-based
> > better than a repository. Quite frankly, momentarily, I think
> > developing such a plugin system is out of scope for most Flink
> > developers. At the current pace of Flink development, collecting these
> > contributions externally without properly maintaining them, doesn't
> > make much sense to me.
> >
> > Cheers,
> > Max
> >
> >
> >
> > On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]>
> wrote:
> > >
> > > Hi,
> > >
> > > many people are building quite exiting stuff on top of Flink. It is
> hard
> > > to keep an good overview on what stuff is available and what not. What
> > > do you think about starting a second git repository "flink-external"
> > > that collects all those code?
> > >
> > > The ideas would be to collect stuff in a central point, such that
> people
> > > can access it easily and get an overview what is already available
> (this
> > > might also avoid duplicate development). It might also be a good point
> > > to show common patterns. In order to collect as much as possible, the
> > > contributing requirement (with respect to testing etc) could be lower
> > > than for Flink itself.
> > >
> > > For example, I recently started a small flink-clojure module with a
> > > simple word-count example to answer a question on SO. Including this in
> > > Flink would not be appropriate. However, for a flink-external repro it
> > > might be nice to have.
> > >
> > > What do you think about it?
> > >
> > >
> > > -Matthias
> > >
> >
>

[Matthias J. Sax-2]

Thanks for the feedback.

I think, the repository does not need to build on a single Flink
release. From my point of view, there should be a single parent module
that contains *independent modules* for each extension/library (there
should be no "cross dependencies" between the modules and each module
can specify the flink dependencies it needs by itself). This make is
most flexible. And if a library works on an old release, it might just
stay there as is. If a library changes (due to Flink changes), it might
just be contained multiple times for different Flink releases.

Each module should provide a short doc (README) that shows how to use an
integrate it with Flink. Thus, the responsibility goes to the
contributor to maintain the library. If it breaks and is not maintained
any further, we can simple remove it.

I agree, that the community might not be able to maintain those
extension/libraries right now. I would put the responsibility (more or
less completely) on the contributor and delete project that do not fix
any more.

@Vasia: a link to a library could be included in the README. If anybody
only wants to share a library but not contribute code, the parent README
could contain a list of additional links.


-Matthias


On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:

> How about, for now, we simply create a page where we gather links/short
> descriptions of all these contributions
> and let the maintenance and dependency management to the tool/library
> creators?
> This way we will at least have these contributions in one place and link to
> them somewhere from the website.
>
> -Vasia.
>
> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>
>> Hi Matthias,
>>
>> Thanks for bringing up this idea. Actually, it has been discussed a
>> couple of times on the mailing list whether we should have a central
>> place for third-party extensions/contributions/libraries. This could
>> either be something package-based or, like you proposed, another
>> repository.
>>
>> An external place for contributions raises a couple of questions
>>
>> - Which version should the external contributions be based on?
>> - How do we make sure, the extensions are continuously updated?
>> (dedicated maintainers or automatic compatibility checks)
>> - How do we easily plug-in the external modules into Flink?
>>
>> In the long term, we really need a solution for these questions. The
>> code base of Flink is growing and more and more packages go to
>> flink-contrib/flink-staging. I would find something packaged-based
>> better than a repository. Quite frankly, momentarily, I think
>> developing such a plugin system is out of scope for most Flink
>> developers. At the current pace of Flink development, collecting these
>> contributions externally without properly maintaining them, doesn't
>> make much sense to me.
>>
>> Cheers,
>> Max
>>
>>
>>
>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> many people are building quite exiting stuff on top of Flink. It is hard
>>> to keep an good overview on what stuff is available and what not. What
>>> do you think about starting a second git repository "flink-external"
>>> that collects all those code?
>>>
>>> The ideas would be to collect stuff in a central point, such that people
>>> can access it easily and get an overview what is already available (this
>>> might also avoid duplicate development). It might also be a good point
>>> to show common patterns. In order to collect as much as possible, the
>>> contributing requirement (with respect to testing etc) could be lower
>>> than for Flink itself.
>>>
>>> For example, I recently started a small flink-clojure module with a
>>> simple word-count example to answer a question on SO. Including this in
>>> Flink would not be appropriate. However, for a flink-external repro it
>>> might be nice to have.
>>>
>>> What do you think about it?
>>>
>>>
>>> -Matthias
>>>
>>
>

[Chiwan Park-2]

+1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.

[1] http://spark-packages.org

> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>
> Thanks for the feedback.
>
> I think, the repository does not need to build on a single Flink
> release. From my point of view, there should be a single parent module
> that contains *independent modules* for each extension/library (there
> should be no "cross dependencies" between the modules and each module
> can specify the flink dependencies it needs by itself). This make is
> most flexible. And if a library works on an old release, it might just
> stay there as is. If a library changes (due to Flink changes), it might
> just be contained multiple times for different Flink releases.
>
> Each module should provide a short doc (README) that shows how to use an
> integrate it with Flink. Thus, the responsibility goes to the
> contributor to maintain the library. If it breaks and is not maintained
> any further, we can simple remove it.
>
> I agree, that the community might not be able to maintain those
> extension/libraries right now. I would put the responsibility (more or
> less completely) on the contributor and delete project that do not fix
> any more.
>
> @Vasia: a link to a library could be included in the README. If anybody
> only wants to share a library but not contribute code, the parent README
> could contain a list of additional links.
>
>
> -Matthias
>
>
> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>> How about, for now, we simply create a page where we gather links/short
>> descriptions of all these contributions
>> and let the maintenance and dependency management to the tool/library
>> creators?
>> This way we will at least have these contributions in one place and link to
>> them somewhere from the website.
>>
>> -Vasia.
>>
>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>
>>> Hi Matthias,
>>>
>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>> couple of times on the mailing list whether we should have a central
>>> place for third-party extensions/contributions/libraries. This could
>>> either be something package-based or, like you proposed, another
>>> repository.
>>>
>>> An external place for contributions raises a couple of questions
>>>
>>> - Which version should the external contributions be based on?
>>> - How do we make sure, the extensions are continuously updated?
>>> (dedicated maintainers or automatic compatibility checks)
>>> - How do we easily plug-in the external modules into Flink?
>>>
>>> In the long term, we really need a solution for these questions. The
>>> code base of Flink is growing and more and more packages go to
>>> flink-contrib/flink-staging. I would find something packaged-based
>>> better than a repository. Quite frankly, momentarily, I think
>>> developing such a plugin system is out of scope for most Flink
>>> developers. At the current pace of Flink development, collecting these
>>> contributions externally without properly maintaining them, doesn't
>>> make much sense to me.
>>>
>>> Cheers,
>>> Max
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>> to keep an good overview on what stuff is available and what not. What
>>>> do you think about starting a second git repository "flink-external"
>>>> that collects all those code?
>>>>
>>>> The ideas would be to collect stuff in a central point, such that people
>>>> can access it easily and get an overview what is already available (this
>>>> might also avoid duplicate development). It might also be a good point
>>>> to show common patterns. In order to collect as much as possible, the
>>>> contributing requirement (with respect to testing etc) could be lower
>>>> than for Flink itself.
>>>>
>>>> For example, I recently started a small flink-clojure module with a
>>>> simple word-count example to answer a question on SO. Including this in
>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>> might be nice to have.
>>>>
>>>> What do you think about it?
>>>>
>>>>
>>>> -Matthias
>>>>
>>>
>>
>

Regards,
Chiwan Park

[mxm]

+1 for your pragmatic approach, Vasia. A simple collection of third
party software using Flink should be enough for now; of course,
outside the Apache realm.

On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]> wrote:

> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
>
> [1] http://spark-packages.org
>
>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>>
>> Thanks for the feedback.
>>
>> I think, the repository does not need to build on a single Flink
>> release. From my point of view, there should be a single parent module
>> that contains *independent modules* for each extension/library (there
>> should be no "cross dependencies" between the modules and each module
>> can specify the flink dependencies it needs by itself). This make is
>> most flexible. And if a library works on an old release, it might just
>> stay there as is. If a library changes (due to Flink changes), it might
>> just be contained multiple times for different Flink releases.
>>
>> Each module should provide a short doc (README) that shows how to use an
>> integrate it with Flink. Thus, the responsibility goes to the
>> contributor to maintain the library. If it breaks and is not maintained
>> any further, we can simple remove it.
>>
>> I agree, that the community might not be able to maintain those
>> extension/libraries right now. I would put the responsibility (more or
>> less completely) on the contributor and delete project that do not fix
>> any more.
>>
>> @Vasia: a link to a library could be included in the README. If anybody
>> only wants to share a library but not contribute code, the parent README
>> could contain a list of additional links.
>>
>>
>> -Matthias
>>
>>
>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>> How about, for now, we simply create a page where we gather links/short
>>> descriptions of all these contributions
>>> and let the maintenance and dependency management to the tool/library
>>> creators?
>>> This way we will at least have these contributions in one place and link to
>>> them somewhere from the website.
>>>
>>> -Vasia.
>>>
>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>>
>>>> Hi Matthias,
>>>>
>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>> couple of times on the mailing list whether we should have a central
>>>> place for third-party extensions/contributions/libraries. This could
>>>> either be something package-based or, like you proposed, another
>>>> repository.
>>>>
>>>> An external place for contributions raises a couple of questions
>>>>
>>>> - Which version should the external contributions be based on?
>>>> - How do we make sure, the extensions are continuously updated?
>>>> (dedicated maintainers or automatic compatibility checks)
>>>> - How do we easily plug-in the external modules into Flink?
>>>>
>>>> In the long term, we really need a solution for these questions. The
>>>> code base of Flink is growing and more and more packages go to
>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>> better than a repository. Quite frankly, momentarily, I think
>>>> developing such a plugin system is out of scope for most Flink
>>>> developers. At the current pace of Flink development, collecting these
>>>> contributions externally without properly maintaining them, doesn't
>>>> make much sense to me.
>>>>
>>>> Cheers,
>>>> Max
>>>>
>>>>
>>>>
>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>> to keep an good overview on what stuff is available and what not. What
>>>>> do you think about starting a second git repository "flink-external"
>>>>> that collects all those code?
>>>>>
>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>> can access it easily and get an overview what is already available (this
>>>>> might also avoid duplicate development). It might also be a good point
>>>>> to show common patterns. In order to collect as much as possible, the
>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>> than for Flink itself.
>>>>>
>>>>> For example, I recently started a small flink-clojure module with a
>>>>> simple word-count example to answer a question on SO. Including this in
>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>> might be nice to have.
>>>>>
>>>>> What do you think about it?
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>
>>>
>>
>
>
>
> Regards,
> Chiwan Park
>
>
>

[Matthias J. Sax-2]

Should we add a new page at Flink project web page?

On 10/08/2015 12:56 PM, Maximilian Michels wrote:

> +1 for your pragmatic approach, Vasia. A simple collection of third
> party software using Flink should be enough for now; of course,
> outside the Apache realm.
>
> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]> wrote:
>> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
>>
>> [1] http://spark-packages.org
>>
>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>>>
>>> Thanks for the feedback.
>>>
>>> I think, the repository does not need to build on a single Flink
>>> release. From my point of view, there should be a single parent module
>>> that contains *independent modules* for each extension/library (there
>>> should be no "cross dependencies" between the modules and each module
>>> can specify the flink dependencies it needs by itself). This make is
>>> most flexible. And if a library works on an old release, it might just
>>> stay there as is. If a library changes (due to Flink changes), it might
>>> just be contained multiple times for different Flink releases.
>>>
>>> Each module should provide a short doc (README) that shows how to use an
>>> integrate it with Flink. Thus, the responsibility goes to the
>>> contributor to maintain the library. If it breaks and is not maintained
>>> any further, we can simple remove it.
>>>
>>> I agree, that the community might not be able to maintain those
>>> extension/libraries right now. I would put the responsibility (more or
>>> less completely) on the contributor and delete project that do not fix
>>> any more.
>>>
>>> @Vasia: a link to a library could be included in the README. If anybody
>>> only wants to share a library but not contribute code, the parent README
>>> could contain a list of additional links.
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>> How about, for now, we simply create a page where we gather links/short
>>>> descriptions of all these contributions
>>>> and let the maintenance and dependency management to the tool/library
>>>> creators?
>>>> This way we will at least have these contributions in one place and link to
>>>> them somewhere from the website.
>>>>
>>>> -Vasia.
>>>>
>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>>>
>>>>> Hi Matthias,
>>>>>
>>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>>> couple of times on the mailing list whether we should have a central
>>>>> place for third-party extensions/contributions/libraries. This could
>>>>> either be something package-based or, like you proposed, another
>>>>> repository.
>>>>>
>>>>> An external place for contributions raises a couple of questions
>>>>>
>>>>> - Which version should the external contributions be based on?
>>>>> - How do we make sure, the extensions are continuously updated?
>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>
>>>>> In the long term, we really need a solution for these questions. The
>>>>> code base of Flink is growing and more and more packages go to
>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>> developing such a plugin system is out of scope for most Flink
>>>>> developers. At the current pace of Flink development, collecting these
>>>>> contributions externally without properly maintaining them, doesn't
>>>>> make much sense to me.
>>>>>
>>>>> Cheers,
>>>>> Max
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>>> to keep an good overview on what stuff is available and what not. What
>>>>>> do you think about starting a second git repository "flink-external"
>>>>>> that collects all those code?
>>>>>>
>>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>>> can access it easily and get an overview what is already available (this
>>>>>> might also avoid duplicate development). It might also be a good point
>>>>>> to show common patterns. In order to collect as much as possible, the
>>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>>> than for Flink itself.
>>>>>>
>>>>>> For example, I recently started a small flink-clojure module with a
>>>>>> simple word-count example to answer a question on SO. Including this in
>>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>>> might be nice to have.
>>>>>>
>>>>>> What do you think about it?
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> Regards,
>> Chiwan Park
>>
>>
>>

[mxm]

IMHO we can do that. There should be a disclaimer that the third party
software is not officially supported.

On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]> wrote:

> Should we add a new page at Flink project web page?
>
> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>> +1 for your pragmatic approach, Vasia. A simple collection of third
>> party software using Flink should be enough for now; of course,
>> outside the Apache realm.
>>
>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]> wrote:
>>> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
>>>
>>> [1] http://spark-packages.org
>>>
>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>>>>
>>>> Thanks for the feedback.
>>>>
>>>> I think, the repository does not need to build on a single Flink
>>>> release. From my point of view, there should be a single parent module
>>>> that contains *independent modules* for each extension/library (there
>>>> should be no "cross dependencies" between the modules and each module
>>>> can specify the flink dependencies it needs by itself). This make is
>>>> most flexible. And if a library works on an old release, it might just
>>>> stay there as is. If a library changes (due to Flink changes), it might
>>>> just be contained multiple times for different Flink releases.
>>>>
>>>> Each module should provide a short doc (README) that shows how to use an
>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>> contributor to maintain the library. If it breaks and is not maintained
>>>> any further, we can simple remove it.
>>>>
>>>> I agree, that the community might not be able to maintain those
>>>> extension/libraries right now. I would put the responsibility (more or
>>>> less completely) on the contributor and delete project that do not fix
>>>> any more.
>>>>
>>>> @Vasia: a link to a library could be included in the README. If anybody
>>>> only wants to share a library but not contribute code, the parent README
>>>> could contain a list of additional links.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>> How about, for now, we simply create a page where we gather links/short
>>>>> descriptions of all these contributions
>>>>> and let the maintenance and dependency management to the tool/library
>>>>> creators?
>>>>> This way we will at least have these contributions in one place and link to
>>>>> them somewhere from the website.
>>>>>
>>>>> -Vasia.
>>>>>
>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>>>>
>>>>>> Hi Matthias,
>>>>>>
>>>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>>>> couple of times on the mailing list whether we should have a central
>>>>>> place for third-party extensions/contributions/libraries. This could
>>>>>> either be something package-based or, like you proposed, another
>>>>>> repository.
>>>>>>
>>>>>> An external place for contributions raises a couple of questions
>>>>>>
>>>>>> - Which version should the external contributions be based on?
>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>
>>>>>> In the long term, we really need a solution for these questions. The
>>>>>> code base of Flink is growing and more and more packages go to
>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>> developers. At the current pace of Flink development, collecting these
>>>>>> contributions externally without properly maintaining them, doesn't
>>>>>> make much sense to me.
>>>>>>
>>>>>> Cheers,
>>>>>> Max
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>>>> to keep an good overview on what stuff is available and what not. What
>>>>>>> do you think about starting a second git repository "flink-external"
>>>>>>> that collects all those code?
>>>>>>>
>>>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>>>> can access it easily and get an overview what is already available (this
>>>>>>> might also avoid duplicate development). It might also be a good point
>>>>>>> to show common patterns. In order to collect as much as possible, the
>>>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>>>> than for Flink itself.
>>>>>>>
>>>>>>> For example, I recently started a small flink-clojure module with a
>>>>>>> simple word-count example to answer a question on SO. Including this in
>>>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>>>> might be nice to have.
>>>>>>>
>>>>>>> What do you think about it?
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> Regards,
>>> Chiwan Park
>>>
>>>
>>>
>

[Matthias J. Sax-2]

Hi,

I just started this. Please see
https://github.com/mjsax/flink-web/tree/flink-external-page

I think, it is the best way to extend the "Downloads" page. I would also
add a link to this on the main page's "Getting Started" section.

As a first try, I started like this:

> Third party packages
>
> This is a list of third party packages (ie, libraries, system extensions, or examples) build for Flink. The Flink community only collects links to those packages but does not maintain them. Thus, they do not belong to the Apache Flink project and the community cannot give any support for them.
> Package Name
>
> Available for Flink 0.8.x and 0.9.x
>
> Short description
>
> Please let us know, if we missed to list your package. Be aware, that we might remove listed packages without notice.

Can you please give me some input, what projects I should add initially?


-Matthias


On 10/08/2015 04:03 PM, Maximilian Michels wrote:

> IMHO we can do that. There should be a disclaimer that the third party
> software is not officially supported.
>
> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]> wrote:
>> Should we add a new page at Flink project web page?
>>
>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>> party software using Flink should be enough for now; of course,
>>> outside the Apache realm.
>>>
>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]> wrote:
>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
>>>>
>>>> [1] http://spark-packages.org
>>>>
>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>>>>>
>>>>> Thanks for the feedback.
>>>>>
>>>>> I think, the repository does not need to build on a single Flink
>>>>> release. From my point of view, there should be a single parent module
>>>>> that contains *independent modules* for each extension/library (there
>>>>> should be no "cross dependencies" between the modules and each module
>>>>> can specify the flink dependencies it needs by itself). This make is
>>>>> most flexible. And if a library works on an old release, it might just
>>>>> stay there as is. If a library changes (due to Flink changes), it might
>>>>> just be contained multiple times for different Flink releases.
>>>>>
>>>>> Each module should provide a short doc (README) that shows how to use an
>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>> contributor to maintain the library. If it breaks and is not maintained
>>>>> any further, we can simple remove it.
>>>>>
>>>>> I agree, that the community might not be able to maintain those
>>>>> extension/libraries right now. I would put the responsibility (more or
>>>>> less completely) on the contributor and delete project that do not fix
>>>>> any more.
>>>>>
>>>>> @Vasia: a link to a library could be included in the README. If anybody
>>>>> only wants to share a library but not contribute code, the parent README
>>>>> could contain a list of additional links.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>> How about, for now, we simply create a page where we gather links/short
>>>>>> descriptions of all these contributions
>>>>>> and let the maintenance and dependency management to the tool/library
>>>>>> creators?
>>>>>> This way we will at least have these contributions in one place and link to
>>>>>> them somewhere from the website.
>>>>>>
>>>>>> -Vasia.
>>>>>>
>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>>>>>
>>>>>>> Hi Matthias,
>>>>>>>
>>>>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>>>>> couple of times on the mailing list whether we should have a central
>>>>>>> place for third-party extensions/contributions/libraries. This could
>>>>>>> either be something package-based or, like you proposed, another
>>>>>>> repository.
>>>>>>>
>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>
>>>>>>> - Which version should the external contributions be based on?
>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>
>>>>>>> In the long term, we really need a solution for these questions. The
>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>> developers. At the current pace of Flink development, collecting these
>>>>>>> contributions externally without properly maintaining them, doesn't
>>>>>>> make much sense to me.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Max
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>>>>> to keep an good overview on what stuff is available and what not. What
>>>>>>>> do you think about starting a second git repository "flink-external"
>>>>>>>> that collects all those code?
>>>>>>>>
>>>>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>>>>> can access it easily and get an overview what is already available (this
>>>>>>>> might also avoid duplicate development). It might also be a good point
>>>>>>>> to show common patterns. In order to collect as much as possible, the
>>>>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>>>>> than for Flink itself.
>>>>>>>>
>>>>>>>> For example, I recently started a small flink-clojure module with a
>>>>>>>> simple word-count example to answer a question on SO. Including this in
>>>>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>>>>> might be nice to have.
>>>>>>>>
>>>>>>>> What do you think about it?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Chiwan Park
>>>>
>>>>
>>>>
>>

[mxm]

Cool. Right now the list is empty. Do you already have a list you
could include in the upcoming pull request? :)

On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]> wrote:

> Hi,
>
> I just started this. Please see
> https://github.com/mjsax/flink-web/tree/flink-external-page
>
> I think, it is the best way to extend the "Downloads" page. I would also
> add a link to this on the main page's "Getting Started" section.
>
> As a first try, I started like this:
>> Third party packages
>>
>> This is a list of third party packages (ie, libraries, system extensions, or examples) build for Flink. The Flink community only collects links to those packages but does not maintain them. Thus, they do not belong to the Apache Flink project and the community cannot give any support for them.
>> Package Name
>>
>> Available for Flink 0.8.x and 0.9.x
>>
>> Short description
>>
>> Please let us know, if we missed to list your package. Be aware, that we might remove listed packages without notice.
>
> Can you please give me some input, what projects I should add initially?
>
>
> -Matthias
>
>
> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>> IMHO we can do that. There should be a disclaimer that the third party
>> software is not officially supported.
>>
>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]> wrote:
>>> Should we add a new page at Flink project web page?
>>>
>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>> party software using Flink should be enough for now; of course,
>>>> outside the Apache realm.
>>>>
>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]> wrote:
>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
>>>>>
>>>>> [1] http://spark-packages.org
>>>>>
>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>
>>>>>> Thanks for the feedback.
>>>>>>
>>>>>> I think, the repository does not need to build on a single Flink
>>>>>> release. From my point of view, there should be a single parent module
>>>>>> that contains *independent modules* for each extension/library (there
>>>>>> should be no "cross dependencies" between the modules and each module
>>>>>> can specify the flink dependencies it needs by itself). This make is
>>>>>> most flexible. And if a library works on an old release, it might just
>>>>>> stay there as is. If a library changes (due to Flink changes), it might
>>>>>> just be contained multiple times for different Flink releases.
>>>>>>
>>>>>> Each module should provide a short doc (README) that shows how to use an
>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>> contributor to maintain the library. If it breaks and is not maintained
>>>>>> any further, we can simple remove it.
>>>>>>
>>>>>> I agree, that the community might not be able to maintain those
>>>>>> extension/libraries right now. I would put the responsibility (more or
>>>>>> less completely) on the contributor and delete project that do not fix
>>>>>> any more.
>>>>>>
>>>>>> @Vasia: a link to a library could be included in the README. If anybody
>>>>>> only wants to share a library but not contribute code, the parent README
>>>>>> could contain a list of additional links.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>> How about, for now, we simply create a page where we gather links/short
>>>>>>> descriptions of all these contributions
>>>>>>> and let the maintenance and dependency management to the tool/library
>>>>>>> creators?
>>>>>>> This way we will at least have these contributions in one place and link to
>>>>>>> them somewhere from the website.
>>>>>>>
>>>>>>> -Vasia.
>>>>>>>
>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>>>>>>
>>>>>>>> Hi Matthias,
>>>>>>>>
>>>>>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>>>>>> couple of times on the mailing list whether we should have a central
>>>>>>>> place for third-party extensions/contributions/libraries. This could
>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>> repository.
>>>>>>>>
>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>
>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>
>>>>>>>> In the long term, we really need a solution for these questions. The
>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>> developers. At the current pace of Flink development, collecting these
>>>>>>>> contributions externally without properly maintaining them, doesn't
>>>>>>>> make much sense to me.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Max
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>>>>>> to keep an good overview on what stuff is available and what not. What
>>>>>>>>> do you think about starting a second git repository "flink-external"
>>>>>>>>> that collects all those code?
>>>>>>>>>
>>>>>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>>>>>> can access it easily and get an overview what is already available (this
>>>>>>>>> might also avoid duplicate development). It might also be a good point
>>>>>>>>> to show common patterns. In order to collect as much as possible, the
>>>>>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>>>>>> than for Flink itself.
>>>>>>>>>
>>>>>>>>> For example, I recently started a small flink-clojure module with a
>>>>>>>>> simple word-count example to answer a question on SO. Including this in
>>>>>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>>>>>> might be nice to have.
>>>>>>>>>
>>>>>>>>> What do you think about it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Chiwan Park
>>>>>
>>>>>
>>>>>
>>>
>

[Matthias J. Sax-2]

I was not sure what we should add and was hoping for input from the
community.

I am aware of the following projects we might want to add:

  - Zeppelin
  - SAMOA
  - Mahout
  - Cascading (dataartisan repo)
  - BigPetStore
  - Gradoop


-Matthias



On 10/09/2015 03:07 PM, Maximilian Michels wrote:

> Cool. Right now the list is empty. Do you already have a list you
> could include in the upcoming pull request? :)
>
> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]> wrote:
>> Hi,
>>
>> I just started this. Please see
>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>
>> I think, it is the best way to extend the "Downloads" page. I would also
>> add a link to this on the main page's "Getting Started" section.
>>
>> As a first try, I started like this:
>>> Third party packages
>>>
>>> This is a list of third party packages (ie, libraries, system extensions, or examples) build for Flink. The Flink community only collects links to those packages but does not maintain them. Thus, they do not belong to the Apache Flink project and the community cannot give any support for them.
>>> Package Name
>>>
>>> Available for Flink 0.8.x and 0.9.x
>>>
>>> Short description
>>>
>>> Please let us know, if we missed to list your package. Be aware, that we might remove listed packages without notice.
>>
>> Can you please give me some input, what projects I should add initially?
>>
>>
>> -Matthias
>>
>>
>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>> IMHO we can do that. There should be a disclaimer that the third party
>>> software is not officially supported.
>>>
>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]> wrote:
>>>> Should we add a new page at Flink project web page?
>>>>
>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>>> party software using Flink should be enough for now; of course,
>>>>> outside the Apache realm.
>>>>>
>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]> wrote:
>>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>
>>>>>> [1] http://spark-packages.org
>>>>>>
>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>>
>>>>>>> Thanks for the feedback.
>>>>>>>
>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>> release. From my point of view, there should be a single parent module
>>>>>>> that contains *independent modules* for each extension/library (there
>>>>>>> should be no "cross dependencies" between the modules and each module
>>>>>>> can specify the flink dependencies it needs by itself). This make is
>>>>>>> most flexible. And if a library works on an old release, it might just
>>>>>>> stay there as is. If a library changes (due to Flink changes), it might
>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>>
>>>>>>> Each module should provide a short doc (README) that shows how to use an
>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>> contributor to maintain the library. If it breaks and is not maintained
>>>>>>> any further, we can simple remove it.
>>>>>>>
>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>> extension/libraries right now. I would put the responsibility (more or
>>>>>>> less completely) on the contributor and delete project that do not fix
>>>>>>> any more.
>>>>>>>
>>>>>>> @Vasia: a link to a library could be included in the README. If anybody
>>>>>>> only wants to share a library but not contribute code, the parent README
>>>>>>> could contain a list of additional links.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>> How about, for now, we simply create a page where we gather links/short
>>>>>>>> descriptions of all these contributions
>>>>>>>> and let the maintenance and dependency management to the tool/library
>>>>>>>> creators?
>>>>>>>> This way we will at least have these contributions in one place and link to
>>>>>>>> them somewhere from the website.
>>>>>>>>
>>>>>>>> -Vasia.
>>>>>>>>
>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]> wrote:
>>>>>>>>
>>>>>>>>> Hi Matthias,
>>>>>>>>>
>>>>>>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>>>>>>> couple of times on the mailing list whether we should have a central
>>>>>>>>> place for third-party extensions/contributions/libraries. This could
>>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>>> repository.
>>>>>>>>>
>>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>>
>>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>>
>>>>>>>>> In the long term, we really need a solution for these questions. The
>>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>>> developers. At the current pace of Flink development, collecting these
>>>>>>>>> contributions externally without properly maintaining them, doesn't
>>>>>>>>> make much sense to me.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Max
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>>>>>>> to keep an good overview on what stuff is available and what not. What
>>>>>>>>>> do you think about starting a second git repository "flink-external"
>>>>>>>>>> that collects all those code?
>>>>>>>>>>
>>>>>>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>>>>>>> can access it easily and get an overview what is already available (this
>>>>>>>>>> might also avoid duplicate development). It might also be a good point
>>>>>>>>>> to show common patterns. In order to collect as much as possible, the
>>>>>>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>>>>>>> than for Flink itself.
>>>>>>>>>>
>>>>>>>>>> For example, I recently started a small flink-clojure module with a
>>>>>>>>>> simple word-count example to answer a question on SO. Including this in
>>>>>>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>>>>>>> might be nice to have.
>>>>>>>>>>
>>>>>>>>>> What do you think about it?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Chiwan Park
>>>>>>
>>>>>>
>>>>>>
>>>>
>>

[Vasiliki Kalavri]

Thank you Matthias!

I'm not sure where the "Downloads" section is the right place for this.
I would actually put it under "Community", with a header "External
Contributions" or something like this, but I'm not feeling strong about
this :)

-Vasia.


On 9 October 2015 at 15:29, Matthias J. Sax <[hidden email]> wrote:

> I was not sure what we should add and was hoping for input from the
> community.
>
> I am aware of the following projects we might want to add:
>
>   - Zeppelin
>   - SAMOA
>   - Mahout
>   - Cascading (dataartisan repo)
>   - BigPetStore
>   - Gradoop
>
>
> -Matthias
>
>
>
> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
> > Cool. Right now the list is empty. Do you already have a list you
> > could include in the upcoming pull request? :)
> >
> > On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]>
> wrote:
> >> Hi,
> >>
> >> I just started this. Please see
> >> https://github.com/mjsax/flink-web/tree/flink-external-page
> >>
> >> I think, it is the best way to extend the "Downloads" page. I would also
> >> add a link to this on the main page's "Getting Started" section.
> >>
> >> As a first try, I started like this:
> >>> Third party packages
> >>>
> >>> This is a list of third party packages (ie, libraries, system
> extensions, or examples) build for Flink. The Flink community only collects
> links to those packages but does not maintain them. Thus, they do not
> belong to the Apache Flink project and the community cannot give any
> support for them.
> >>> Package Name
> >>>
> >>> Available for Flink 0.8.x and 0.9.x
> >>>
> >>> Short description
> >>>
> >>> Please let us know, if we missed to list your package. Be aware, that
> we might remove listed packages without notice.
> >>
> >> Can you please give me some input, what projects I should add initially?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
> >>> IMHO we can do that. There should be a disclaimer that the third party
> >>> software is not officially supported.
> >>>
> >>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]>
> wrote:
> >>>> Should we add a new page at Flink project web page?
> >>>>
> >>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
> >>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
> >>>>> party software using Flink should be enough for now; of course,
> >>>>> outside the Apache realm.
> >>>>>
> >>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]>
> wrote:
> >>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
> like Spark Packages [1] would be helpful to manage external contribution.
> >>>>>>
> >>>>>> [1] http://spark-packages.org
> >>>>>>
> >>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]>
> wrote:
> >>>>>>>
> >>>>>>> Thanks for the feedback.
> >>>>>>>
> >>>>>>> I think, the repository does not need to build on a single Flink
> >>>>>>> release. From my point of view, there should be a single parent
> module
> >>>>>>> that contains *independent modules* for each extension/library
> (there
> >>>>>>> should be no "cross dependencies" between the modules and each
> module
> >>>>>>> can specify the flink dependencies it needs by itself). This make
> is
> >>>>>>> most flexible. And if a library works on an old release, it might
> just
> >>>>>>> stay there as is. If a library changes (due to Flink changes), it
> might
> >>>>>>> just be contained multiple times for different Flink releases.
> >>>>>>>
> >>>>>>> Each module should provide a short doc (README) that shows how to
> use an
> >>>>>>> integrate it with Flink. Thus, the responsibility goes to the
> >>>>>>> contributor to maintain the library. If it breaks and is not
> maintained
> >>>>>>> any further, we can simple remove it.
> >>>>>>>
> >>>>>>> I agree, that the community might not be able to maintain those
> >>>>>>> extension/libraries right now. I would put the responsibility
> (more or
> >>>>>>> less completely) on the contributor and delete project that do not
> fix
> >>>>>>> any more.
> >>>>>>>
> >>>>>>> @Vasia: a link to a library could be included in the README. If
> anybody
> >>>>>>> only wants to share a library but not contribute code, the parent
> README
> >>>>>>> could contain a list of additional links.
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
> >>>>>>>> How about, for now, we simply create a page where we gather
> links/short
> >>>>>>>> descriptions of all these contributions
> >>>>>>>> and let the maintenance and dependency management to the
> tool/library
> >>>>>>>> creators?
> >>>>>>>> This way we will at least have these contributions in one place
> and link to
> >>>>>>>> them somewhere from the website.
> >>>>>>>>
> >>>>>>>> -Vasia.
> >>>>>>>>
> >>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]>
> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Matthias,
> >>>>>>>>>
> >>>>>>>>> Thanks for bringing up this idea. Actually, it has been
> discussed a
> >>>>>>>>> couple of times on the mailing list whether we should have a
> central
> >>>>>>>>> place for third-party extensions/contributions/libraries. This
> could
> >>>>>>>>> either be something package-based or, like you proposed, another
> >>>>>>>>> repository.
> >>>>>>>>>
> >>>>>>>>> An external place for contributions raises a couple of questions
> >>>>>>>>>
> >>>>>>>>> - Which version should the external contributions be based on?
> >>>>>>>>> - How do we make sure, the extensions are continuously updated?
> >>>>>>>>> (dedicated maintainers or automatic compatibility checks)
> >>>>>>>>> - How do we easily plug-in the external modules into Flink?
> >>>>>>>>>
> >>>>>>>>> In the long term, we really need a solution for these questions.
> The
> >>>>>>>>> code base of Flink is growing and more and more packages go to
> >>>>>>>>> flink-contrib/flink-staging. I would find something
> packaged-based
> >>>>>>>>> better than a repository. Quite frankly, momentarily, I think
> >>>>>>>>> developing such a plugin system is out of scope for most Flink
> >>>>>>>>> developers. At the current pace of Flink development, collecting
> these
> >>>>>>>>> contributions externally without properly maintaining them,
> doesn't
> >>>>>>>>> make much sense to me.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Max
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <
> [hidden email]> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> many people are building quite exiting stuff on top of Flink.
> It is hard
> >>>>>>>>>> to keep an good overview on what stuff is available and what
> not. What
> >>>>>>>>>> do you think about starting a second git repository
> "flink-external"
> >>>>>>>>>> that collects all those code?
> >>>>>>>>>>
> >>>>>>>>>> The ideas would be to collect stuff in a central point, such
> that people
> >>>>>>>>>> can access it easily and get an overview what is already
> available (this
> >>>>>>>>>> might also avoid duplicate development). It might also be a
> good point
> >>>>>>>>>> to show common patterns. In order to collect as much as
> possible, the
> >>>>>>>>>> contributing requirement (with respect to testing etc) could be
> lower
> >>>>>>>>>> than for Flink itself.
> >>>>>>>>>>
> >>>>>>>>>> For example, I recently started a small flink-clojure module
> with a
> >>>>>>>>>> simple word-count example to answer a question on SO. Including
> this in
> >>>>>>>>>> Flink would not be appropriate. However, for a flink-external
> repro it
> >>>>>>>>>> might be nice to have.
> >>>>>>>>>>
> >>>>>>>>>> What do you think about it?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Regards,
> >>>>>> Chiwan Park
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>
>
>

[mxm]

Yes, Community is a better place. You can also add the Dataflow Runner
https://github.com/dataArtisans/flink-dataflow.

On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
<[hidden email]> wrote:

> Thank you Matthias!
>
> I'm not sure where the "Downloads" section is the right place for this.
> I would actually put it under "Community", with a header "External
> Contributions" or something like this, but I'm not feeling strong about
> this :)
>
> -Vasia.
>
>
> On 9 October 2015 at 15:29, Matthias J. Sax <[hidden email]> wrote:
>
>> I was not sure what we should add and was hoping for input from the
>> community.
>>
>> I am aware of the following projects we might want to add:
>>
>>   - Zeppelin
>>   - SAMOA
>>   - Mahout
>>   - Cascading (dataartisan repo)
>>   - BigPetStore
>>   - Gradoop
>>
>>
>> -Matthias
>>
>>
>>
>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>> > Cool. Right now the list is empty. Do you already have a list you
>> > could include in the upcoming pull request? :)
>> >
>> > On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]>
>> wrote:
>> >> Hi,
>> >>
>> >> I just started this. Please see
>> >> https://github.com/mjsax/flink-web/tree/flink-external-page
>> >>
>> >> I think, it is the best way to extend the "Downloads" page. I would also
>> >> add a link to this on the main page's "Getting Started" section.
>> >>
>> >> As a first try, I started like this:
>> >>> Third party packages
>> >>>
>> >>> This is a list of third party packages (ie, libraries, system
>> extensions, or examples) build for Flink. The Flink community only collects
>> links to those packages but does not maintain them. Thus, they do not
>> belong to the Apache Flink project and the community cannot give any
>> support for them.
>> >>> Package Name
>> >>>
>> >>> Available for Flink 0.8.x and 0.9.x
>> >>>
>> >>> Short description
>> >>>
>> >>> Please let us know, if we missed to list your package. Be aware, that
>> we might remove listed packages without notice.
>> >>
>> >> Can you please give me some input, what projects I should add initially?
>> >>
>> >>
>> >> -Matthias
>> >>
>> >>
>> >> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>> >>> IMHO we can do that. There should be a disclaimer that the third party
>> >>> software is not officially supported.
>> >>>
>> >>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]>
>> wrote:
>> >>>> Should we add a new page at Flink project web page?
>> >>>>
>> >>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>> >>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>> >>>>> party software using Flink should be enough for now; of course,
>> >>>>> outside the Apache realm.
>> >>>>>
>> >>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]>
>> wrote:
>> >>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
>> like Spark Packages [1] would be helpful to manage external contribution.
>> >>>>>>
>> >>>>>> [1] http://spark-packages.org
>> >>>>>>
>> >>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]>
>> wrote:
>> >>>>>>>
>> >>>>>>> Thanks for the feedback.
>> >>>>>>>
>> >>>>>>> I think, the repository does not need to build on a single Flink
>> >>>>>>> release. From my point of view, there should be a single parent
>> module
>> >>>>>>> that contains *independent modules* for each extension/library
>> (there
>> >>>>>>> should be no "cross dependencies" between the modules and each
>> module
>> >>>>>>> can specify the flink dependencies it needs by itself). This make
>> is
>> >>>>>>> most flexible. And if a library works on an old release, it might
>> just
>> >>>>>>> stay there as is. If a library changes (due to Flink changes), it
>> might
>> >>>>>>> just be contained multiple times for different Flink releases.
>> >>>>>>>
>> >>>>>>> Each module should provide a short doc (README) that shows how to
>> use an
>> >>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>> >>>>>>> contributor to maintain the library. If it breaks and is not
>> maintained
>> >>>>>>> any further, we can simple remove it.
>> >>>>>>>
>> >>>>>>> I agree, that the community might not be able to maintain those
>> >>>>>>> extension/libraries right now. I would put the responsibility
>> (more or
>> >>>>>>> less completely) on the contributor and delete project that do not
>> fix
>> >>>>>>> any more.
>> >>>>>>>
>> >>>>>>> @Vasia: a link to a library could be included in the README. If
>> anybody
>> >>>>>>> only wants to share a library but not contribute code, the parent
>> README
>> >>>>>>> could contain a list of additional links.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -Matthias
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>> >>>>>>>> How about, for now, we simply create a page where we gather
>> links/short
>> >>>>>>>> descriptions of all these contributions
>> >>>>>>>> and let the maintenance and dependency management to the
>> tool/library
>> >>>>>>>> creators?
>> >>>>>>>> This way we will at least have these contributions in one place
>> and link to
>> >>>>>>>> them somewhere from the website.
>> >>>>>>>>
>> >>>>>>>> -Vasia.
>> >>>>>>>>
>> >>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]>
>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi Matthias,
>> >>>>>>>>>
>> >>>>>>>>> Thanks for bringing up this idea. Actually, it has been
>> discussed a
>> >>>>>>>>> couple of times on the mailing list whether we should have a
>> central
>> >>>>>>>>> place for third-party extensions/contributions/libraries. This
>> could
>> >>>>>>>>> either be something package-based or, like you proposed, another
>> >>>>>>>>> repository.
>> >>>>>>>>>
>> >>>>>>>>> An external place for contributions raises a couple of questions
>> >>>>>>>>>
>> >>>>>>>>> - Which version should the external contributions be based on?
>> >>>>>>>>> - How do we make sure, the extensions are continuously updated?
>> >>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>> >>>>>>>>> - How do we easily plug-in the external modules into Flink?
>> >>>>>>>>>
>> >>>>>>>>> In the long term, we really need a solution for these questions.
>> The
>> >>>>>>>>> code base of Flink is growing and more and more packages go to
>> >>>>>>>>> flink-contrib/flink-staging. I would find something
>> packaged-based
>> >>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>> >>>>>>>>> developing such a plugin system is out of scope for most Flink
>> >>>>>>>>> developers. At the current pace of Flink development, collecting
>> these
>> >>>>>>>>> contributions externally without properly maintaining them,
>> doesn't
>> >>>>>>>>> make much sense to me.
>> >>>>>>>>>
>> >>>>>>>>> Cheers,
>> >>>>>>>>> Max
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <
>> [hidden email]> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Hi,
>> >>>>>>>>>>
>> >>>>>>>>>> many people are building quite exiting stuff on top of Flink.
>> It is hard
>> >>>>>>>>>> to keep an good overview on what stuff is available and what
>> not. What
>> >>>>>>>>>> do you think about starting a second git repository
>> "flink-external"
>> >>>>>>>>>> that collects all those code?
>> >>>>>>>>>>
>> >>>>>>>>>> The ideas would be to collect stuff in a central point, such
>> that people
>> >>>>>>>>>> can access it easily and get an overview what is already
>> available (this
>> >>>>>>>>>> might also avoid duplicate development). It might also be a
>> good point
>> >>>>>>>>>> to show common patterns. In order to collect as much as
>> possible, the
>> >>>>>>>>>> contributing requirement (with respect to testing etc) could be
>> lower
>> >>>>>>>>>> than for Flink itself.
>> >>>>>>>>>>
>> >>>>>>>>>> For example, I recently started a small flink-clojure module
>> with a
>> >>>>>>>>>> simple word-count example to answer a question on SO. Including
>> this in
>> >>>>>>>>>> Flink would not be appropriate. However, for a flink-external
>> repro it
>> >>>>>>>>>> might be nice to have.
>> >>>>>>>>>>
>> >>>>>>>>>> What do you think about it?
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> -Matthias
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Regards,
>> >>>>>> Chiwan Park
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>
>>
>>

[Matthias J. Sax-2]

Hi,

I updated the flink-external section on the Flink Web-Page:
https://github.com/mjsax/flink-web/tree/flink-external-page

The section is now located in "Contribute" page. The layout needs some
refinement though... Some Project are "previews", ie, Flink support was
announced but there is not information on the according project web
pages. We might want to reach out to those people to see if we should
include those project already or just add them later on.

Please give feedback.


-Matthias




On 10/09/2015 03:34 PM, Maximilian Michels wrote:

> Yes, Community is a better place. You can also add the Dataflow Runner
> https://github.com/dataArtisans/flink-dataflow.
>
> On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
> <[hidden email]> wrote:
>> Thank you Matthias!
>>
>> I'm not sure where the "Downloads" section is the right place for this.
>> I would actually put it under "Community", with a header "External
>> Contributions" or something like this, but I'm not feeling strong about
>> this :)
>>
>> -Vasia.
>>
>>
>> On 9 October 2015 at 15:29, Matthias J. Sax <[hidden email]> wrote:
>>
>>> I was not sure what we should add and was hoping for input from the
>>> community.
>>>
>>> I am aware of the following projects we might want to add:
>>>
>>>   - Zeppelin
>>>   - SAMOA
>>>   - Mahout
>>>   - Cascading (dataartisan repo)
>>>   - BigPetStore
>>>   - Gradoop
>>>
>>>
>>> -Matthias
>>>
>>>
>>>
>>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>>>> Cool. Right now the list is empty. Do you already have a list you
>>>> could include in the upcoming pull request? :)
>>>>
>>>> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]>
>>> wrote:
>>>>> Hi,
>>>>>
>>>>> I just started this. Please see
>>>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>>>
>>>>> I think, it is the best way to extend the "Downloads" page. I would also
>>>>> add a link to this on the main page's "Getting Started" section.
>>>>>
>>>>> As a first try, I started like this:
>>>>>> Third party packages
>>>>>>
>>>>>> This is a list of third party packages (ie, libraries, system
>>> extensions, or examples) build for Flink. The Flink community only collects
>>> links to those packages but does not maintain them. Thus, they do not
>>> belong to the Apache Flink project and the community cannot give any
>>> support for them.
>>>>>> Package Name
>>>>>>
>>>>>> Available for Flink 0.8.x and 0.9.x
>>>>>>
>>>>>> Short description
>>>>>>
>>>>>> Please let us know, if we missed to list your package. Be aware, that
>>> we might remove listed packages without notice.
>>>>>
>>>>> Can you please give me some input, what projects I should add initially?
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>>>>> IMHO we can do that. There should be a disclaimer that the third party
>>>>>> software is not officially supported.
>>>>>>
>>>>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]>
>>> wrote:
>>>>>>> Should we add a new page at Flink project web page?
>>>>>>>
>>>>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>>>>>> party software using Flink should be enough for now; of course,
>>>>>>>> outside the Apache realm.
>>>>>>>>
>>>>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]>
>>> wrote:
>>>>>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
>>> like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>>>>
>>>>>>>>> [1] http://spark-packages.org
>>>>>>>>>
>>>>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]>
>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>
>>>>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>>>>> release. From my point of view, there should be a single parent
>>> module
>>>>>>>>>> that contains *independent modules* for each extension/library
>>> (there
>>>>>>>>>> should be no "cross dependencies" between the modules and each
>>> module
>>>>>>>>>> can specify the flink dependencies it needs by itself). This make
>>> is
>>>>>>>>>> most flexible. And if a library works on an old release, it might
>>> just
>>>>>>>>>> stay there as is. If a library changes (due to Flink changes), it
>>> might
>>>>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>>>>>
>>>>>>>>>> Each module should provide a short doc (README) that shows how to
>>> use an
>>>>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>>>>> contributor to maintain the library. If it breaks and is not
>>> maintained
>>>>>>>>>> any further, we can simple remove it.
>>>>>>>>>>
>>>>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>>>>> extension/libraries right now. I would put the responsibility
>>> (more or
>>>>>>>>>> less completely) on the contributor and delete project that do not
>>> fix
>>>>>>>>>> any more.
>>>>>>>>>>
>>>>>>>>>> @Vasia: a link to a library could be included in the README. If
>>> anybody
>>>>>>>>>> only wants to share a library but not contribute code, the parent
>>> README
>>>>>>>>>> could contain a list of additional links.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>>>>> How about, for now, we simply create a page where we gather
>>> links/short
>>>>>>>>>>> descriptions of all these contributions
>>>>>>>>>>> and let the maintenance and dependency management to the
>>> tool/library
>>>>>>>>>>> creators?
>>>>>>>>>>> This way we will at least have these contributions in one place
>>> and link to
>>>>>>>>>>> them somewhere from the website.
>>>>>>>>>>>
>>>>>>>>>>> -Vasia.
>>>>>>>>>>>
>>>>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]>
>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for bringing up this idea. Actually, it has been
>>> discussed a
>>>>>>>>>>>> couple of times on the mailing list whether we should have a
>>> central
>>>>>>>>>>>> place for third-party extensions/contributions/libraries. This
>>> could
>>>>>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>>>>>> repository.
>>>>>>>>>>>>
>>>>>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>>>>>
>>>>>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>>>>>
>>>>>>>>>>>> In the long term, we really need a solution for these questions.
>>> The
>>>>>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>>>>>> flink-contrib/flink-staging. I would find something
>>> packaged-based
>>>>>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>>>>>> developers. At the current pace of Flink development, collecting
>>> these
>>>>>>>>>>>> contributions externally without properly maintaining them,
>>> doesn't
>>>>>>>>>>>> make much sense to me.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Max
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <
>>> [hidden email]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> many people are building quite exiting stuff on top of Flink.
>>> It is hard
>>>>>>>>>>>>> to keep an good overview on what stuff is available and what
>>> not. What
>>>>>>>>>>>>> do you think about starting a second git repository
>>> "flink-external"
>>>>>>>>>>>>> that collects all those code?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The ideas would be to collect stuff in a central point, such
>>> that people
>>>>>>>>>>>>> can access it easily and get an overview what is already
>>> available (this
>>>>>>>>>>>>> might also avoid duplicate development). It might also be a
>>> good point
>>>>>>>>>>>>> to show common patterns. In order to collect as much as
>>> possible, the
>>>>>>>>>>>>> contributing requirement (with respect to testing etc) could be
>>> lower
>>>>>>>>>>>>> than for Flink itself.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example, I recently started a small flink-clojure module
>>> with a
>>>>>>>>>>>>> simple word-count example to answer a question on SO. Including
>>> this in
>>>>>>>>>>>>> Flink would not be appropriate. However, for a flink-external
>>> repro it
>>>>>>>>>>>>> might be nice to have.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Chiwan Park
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>>>

[Matthias J. Sax-2]

Just updated this. Improved the layout and added FastR project.

https://github.com/mjsax/flink-web/tree/flink-external-page

-Matthias

On 10/27/2015 03:25 AM, Matthias J. Sax wrote:

> Hi,
>
> I updated the flink-external section on the Flink Web-Page:
> https://github.com/mjsax/flink-web/tree/flink-external-page
>
> The section is now located in "Contribute" page. The layout needs some
> refinement though... Some Project are "previews", ie, Flink support was
> announced but there is not information on the according project web
> pages. We might want to reach out to those people to see if we should
> include those project already or just add them later on.
>
> Please give feedback.
>
>
> -Matthias
>
>
>
>
> On 10/09/2015 03:34 PM, Maximilian Michels wrote:
>> Yes, Community is a better place. You can also add the Dataflow Runner
>> https://github.com/dataArtisans/flink-dataflow.
>>
>> On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
>> <[hidden email]> wrote:
>>> Thank you Matthias!
>>>
>>> I'm not sure where the "Downloads" section is the right place for this.
>>> I would actually put it under "Community", with a header "External
>>> Contributions" or something like this, but I'm not feeling strong about
>>> this :)
>>>
>>> -Vasia.
>>>
>>>
>>> On 9 October 2015 at 15:29, Matthias J. Sax <[hidden email]> wrote:
>>>
>>>> I was not sure what we should add and was hoping for input from the
>>>> community.
>>>>
>>>> I am aware of the following projects we might want to add:
>>>>
>>>>   - Zeppelin
>>>>   - SAMOA
>>>>   - Mahout
>>>>   - Cascading (dataartisan repo)
>>>>   - BigPetStore
>>>>   - Gradoop
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>>
>>>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>>>>> Cool. Right now the list is empty. Do you already have a list you
>>>>> could include in the upcoming pull request? :)
>>>>>
>>>>> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]>
>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I just started this. Please see
>>>>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>>>>
>>>>>> I think, it is the best way to extend the "Downloads" page. I would also
>>>>>> add a link to this on the main page's "Getting Started" section.
>>>>>>
>>>>>> As a first try, I started like this:
>>>>>>> Third party packages
>>>>>>>
>>>>>>> This is a list of third party packages (ie, libraries, system
>>>> extensions, or examples) build for Flink. The Flink community only collects
>>>> links to those packages but does not maintain them. Thus, they do not
>>>> belong to the Apache Flink project and the community cannot give any
>>>> support for them.
>>>>>>> Package Name
>>>>>>>
>>>>>>> Available for Flink 0.8.x and 0.9.x
>>>>>>>
>>>>>>> Short description
>>>>>>>
>>>>>>> Please let us know, if we missed to list your package. Be aware, that
>>>> we might remove listed packages without notice.
>>>>>>
>>>>>> Can you please give me some input, what projects I should add initially?
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>>>>>> IMHO we can do that. There should be a disclaimer that the third party
>>>>>>> software is not officially supported.
>>>>>>>
>>>>>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]>
>>>> wrote:
>>>>>>>> Should we add a new page at Flink project web page?
>>>>>>>>
>>>>>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>>>>>>> party software using Flink should be enough for now; of course,
>>>>>>>>> outside the Apache realm.
>>>>>>>>>
>>>>>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]>
>>>> wrote:
>>>>>>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
>>>> like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>>>>>
>>>>>>>>>> [1] http://spark-packages.org
>>>>>>>>>>
>>>>>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>
>>>>>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>>>>>> release. From my point of view, there should be a single parent
>>>> module
>>>>>>>>>>> that contains *independent modules* for each extension/library
>>>> (there
>>>>>>>>>>> should be no "cross dependencies" between the modules and each
>>>> module
>>>>>>>>>>> can specify the flink dependencies it needs by itself). This make
>>>> is
>>>>>>>>>>> most flexible. And if a library works on an old release, it might
>>>> just
>>>>>>>>>>> stay there as is. If a library changes (due to Flink changes), it
>>>> might
>>>>>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>>>>>>
>>>>>>>>>>> Each module should provide a short doc (README) that shows how to
>>>> use an
>>>>>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>>>>>> contributor to maintain the library. If it breaks and is not
>>>> maintained
>>>>>>>>>>> any further, we can simple remove it.
>>>>>>>>>>>
>>>>>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>>>>>> extension/libraries right now. I would put the responsibility
>>>> (more or
>>>>>>>>>>> less completely) on the contributor and delete project that do not
>>>> fix
>>>>>>>>>>> any more.
>>>>>>>>>>>
>>>>>>>>>>> @Vasia: a link to a library could be included in the README. If
>>>> anybody
>>>>>>>>>>> only wants to share a library but not contribute code, the parent
>>>> README
>>>>>>>>>>> could contain a list of additional links.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>>>>>> How about, for now, we simply create a page where we gather
>>>> links/short
>>>>>>>>>>>> descriptions of all these contributions
>>>>>>>>>>>> and let the maintenance and dependency management to the
>>>> tool/library
>>>>>>>>>>>> creators?
>>>>>>>>>>>> This way we will at least have these contributions in one place
>>>> and link to
>>>>>>>>>>>> them somewhere from the website.
>>>>>>>>>>>>
>>>>>>>>>>>> -Vasia.
>>>>>>>>>>>>
>>>>>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for bringing up this idea. Actually, it has been
>>>> discussed a
>>>>>>>>>>>>> couple of times on the mailing list whether we should have a
>>>> central
>>>>>>>>>>>>> place for third-party extensions/contributions/libraries. This
>>>> could
>>>>>>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>>>>>>> repository.
>>>>>>>>>>>>>
>>>>>>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the long term, we really need a solution for these questions.
>>>> The
>>>>>>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>>>>>>> flink-contrib/flink-staging. I would find something
>>>> packaged-based
>>>>>>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>>>>>>> developers. At the current pace of Flink development, collecting
>>>> these
>>>>>>>>>>>>> contributions externally without properly maintaining them,
>>>> doesn't
>>>>>>>>>>>>> make much sense to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Max
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <
>>>> [hidden email]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> many people are building quite exiting stuff on top of Flink.
>>>> It is hard
>>>>>>>>>>>>>> to keep an good overview on what stuff is available and what
>>>> not. What
>>>>>>>>>>>>>> do you think about starting a second git repository
>>>> "flink-external"
>>>>>>>>>>>>>> that collects all those code?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The ideas would be to collect stuff in a central point, such
>>>> that people
>>>>>>>>>>>>>> can access it easily and get an overview what is already
>>>> available (this
>>>>>>>>>>>>>> might also avoid duplicate development). It might also be a
>>>> good point
>>>>>>>>>>>>>> to show common patterns. In order to collect as much as
>>>> possible, the
>>>>>>>>>>>>>> contributing requirement (with respect to testing etc) could be
>>>> lower
>>>>>>>>>>>>>> than for Flink itself.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example, I recently started a small flink-clojure module
>>>> with a
>>>>>>>>>>>>>> simple word-count example to answer a question on SO. Including
>>>> this in
>>>>>>>>>>>>>> Flink would not be appropriate. However, for a flink-external
>>>> repro it
>>>>>>>>>>>>>> might be nice to have.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Chiwan Park
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>>
>

[mxm]

Thanks Matthias! I made a comment. Please open a pull request.

On Tue, Oct 27, 2015 at 10:37 PM, Matthias J. Sax <[hidden email]> wrote:

> Just updated this. Improved the layout and added FastR project.
>
> https://github.com/mjsax/flink-web/tree/flink-external-page
>
> -Matthias
>
> On 10/27/2015 03:25 AM, Matthias J. Sax wrote:
>> Hi,
>>
>> I updated the flink-external section on the Flink Web-Page:
>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>
>> The section is now located in "Contribute" page. The layout needs some
>> refinement though... Some Project are "previews", ie, Flink support was
>> announced but there is not information on the according project web
>> pages. We might want to reach out to those people to see if we should
>> include those project already or just add them later on.
>>
>> Please give feedback.
>>
>>
>> -Matthias
>>
>>
>>
>>
>> On 10/09/2015 03:34 PM, Maximilian Michels wrote:
>>> Yes, Community is a better place. You can also add the Dataflow Runner
>>> https://github.com/dataArtisans/flink-dataflow.
>>>
>>> On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
>>> <[hidden email]> wrote:
>>>> Thank you Matthias!
>>>>
>>>> I'm not sure where the "Downloads" section is the right place for this.
>>>> I would actually put it under "Community", with a header "External
>>>> Contributions" or something like this, but I'm not feeling strong about
>>>> this :)
>>>>
>>>> -Vasia.
>>>>
>>>>
>>>> On 9 October 2015 at 15:29, Matthias J. Sax <[hidden email]> wrote:
>>>>
>>>>> I was not sure what we should add and was hoping for input from the
>>>>> community.
>>>>>
>>>>> I am aware of the following projects we might want to add:
>>>>>
>>>>>   - Zeppelin
>>>>>   - SAMOA
>>>>>   - Mahout
>>>>>   - Cascading (dataartisan repo)
>>>>>   - BigPetStore
>>>>>   - Gradoop
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>>
>>>>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>>>>>> Cool. Right now the list is empty. Do you already have a list you
>>>>>> could include in the upcoming pull request? :)
>>>>>>
>>>>>> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]>
>>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I just started this. Please see
>>>>>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>>>>>
>>>>>>> I think, it is the best way to extend the "Downloads" page. I would also
>>>>>>> add a link to this on the main page's "Getting Started" section.
>>>>>>>
>>>>>>> As a first try, I started like this:
>>>>>>>> Third party packages
>>>>>>>>
>>>>>>>> This is a list of third party packages (ie, libraries, system
>>>>> extensions, or examples) build for Flink. The Flink community only collects
>>>>> links to those packages but does not maintain them. Thus, they do not
>>>>> belong to the Apache Flink project and the community cannot give any
>>>>> support for them.
>>>>>>>> Package Name
>>>>>>>>
>>>>>>>> Available for Flink 0.8.x and 0.9.x
>>>>>>>>
>>>>>>>> Short description
>>>>>>>>
>>>>>>>> Please let us know, if we missed to list your package. Be aware, that
>>>>> we might remove listed packages without notice.
>>>>>>>
>>>>>>> Can you please give me some input, what projects I should add initially?
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>>>>>>> IMHO we can do that. There should be a disclaimer that the third party
>>>>>>>> software is not officially supported.
>>>>>>>>
>>>>>>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]>
>>>>> wrote:
>>>>>>>>> Should we add a new page at Flink project web page?
>>>>>>>>>
>>>>>>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>>>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>>>>>>>> party software using Flink should be enough for now; of course,
>>>>>>>>>> outside the Apache realm.
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]>
>>>>> wrote:
>>>>>>>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
>>>>> like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>>>>>>
>>>>>>>>>>> [1] http://spark-packages.org
>>>>>>>>>>>
>>>>>>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]>
>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>
>>>>>>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>>>>>>> release. From my point of view, there should be a single parent
>>>>> module
>>>>>>>>>>>> that contains *independent modules* for each extension/library
>>>>> (there
>>>>>>>>>>>> should be no "cross dependencies" between the modules and each
>>>>> module
>>>>>>>>>>>> can specify the flink dependencies it needs by itself). This make
>>>>> is
>>>>>>>>>>>> most flexible. And if a library works on an old release, it might
>>>>> just
>>>>>>>>>>>> stay there as is. If a library changes (due to Flink changes), it
>>>>> might
>>>>>>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>>>>>>>
>>>>>>>>>>>> Each module should provide a short doc (README) that shows how to
>>>>> use an
>>>>>>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>>>>>>> contributor to maintain the library. If it breaks and is not
>>>>> maintained
>>>>>>>>>>>> any further, we can simple remove it.
>>>>>>>>>>>>
>>>>>>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>>>>>>> extension/libraries right now. I would put the responsibility
>>>>> (more or
>>>>>>>>>>>> less completely) on the contributor and delete project that do not
>>>>> fix
>>>>>>>>>>>> any more.
>>>>>>>>>>>>
>>>>>>>>>>>> @Vasia: a link to a library could be included in the README. If
>>>>> anybody
>>>>>>>>>>>> only wants to share a library but not contribute code, the parent
>>>>> README
>>>>>>>>>>>> could contain a list of additional links.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>>>>>>> How about, for now, we simply create a page where we gather
>>>>> links/short
>>>>>>>>>>>>> descriptions of all these contributions
>>>>>>>>>>>>> and let the maintenance and dependency management to the
>>>>> tool/library
>>>>>>>>>>>>> creators?
>>>>>>>>>>>>> This way we will at least have these contributions in one place
>>>>> and link to
>>>>>>>>>>>>> them somewhere from the website.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Vasia.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]>
>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for bringing up this idea. Actually, it has been
>>>>> discussed a
>>>>>>>>>>>>>> couple of times on the mailing list whether we should have a
>>>>> central
>>>>>>>>>>>>>> place for third-party extensions/contributions/libraries. This
>>>>> could
>>>>>>>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>>>>>>>> repository.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the long term, we really need a solution for these questions.
>>>>> The
>>>>>>>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>>>>>>>> flink-contrib/flink-staging. I would find something
>>>>> packaged-based
>>>>>>>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>>>>>>>> developers. At the current pace of Flink development, collecting
>>>>> these
>>>>>>>>>>>>>> contributions externally without properly maintaining them,
>>>>> doesn't
>>>>>>>>>>>>>> make much sense to me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <
>>>>> [hidden email]> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> many people are building quite exiting stuff on top of Flink.
>>>>> It is hard
>>>>>>>>>>>>>>> to keep an good overview on what stuff is available and what
>>>>> not. What
>>>>>>>>>>>>>>> do you think about starting a second git repository
>>>>> "flink-external"
>>>>>>>>>>>>>>> that collects all those code?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The ideas would be to collect stuff in a central point, such
>>>>> that people
>>>>>>>>>>>>>>> can access it easily and get an overview what is already
>>>>> available (this
>>>>>>>>>>>>>>> might also avoid duplicate development). It might also be a
>>>>> good point
>>>>>>>>>>>>>>> to show common patterns. In order to collect as much as
>>>>> possible, the
>>>>>>>>>>>>>>> contributing requirement (with respect to testing etc) could be
>>>>> lower
>>>>>>>>>>>>>>> than for Flink itself.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For example, I recently started a small flink-clojure module
>>>>> with a
>>>>>>>>>>>>>>> simple word-count example to answer a question on SO. Including
>>>>> this in
>>>>>>>>>>>>>>> Flink would not be appropriate. However, for a flink-external
>>>>> repro it
>>>>>>>>>>>>>>> might be nice to have.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Chiwan Park
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>
>

[Matthias J. Sax-2]

What is the status here: https://github.com/apache/flink-web/pull/12

Please give feedback.

On 10/28/2015 11:02 AM, Maximilian Michels wrote:

> Thanks Matthias! I made a comment. Please open a pull request.
>
> On Tue, Oct 27, 2015 at 10:37 PM, Matthias J. Sax <[hidden email]> wrote:
>> Just updated this. Improved the layout and added FastR project.
>>
>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>
>> -Matthias
>>
>> On 10/27/2015 03:25 AM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> I updated the flink-external section on the Flink Web-Page:
>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>
>>> The section is now located in "Contribute" page. The layout needs some
>>> refinement though... Some Project are "previews", ie, Flink support was
>>> announced but there is not information on the according project web
>>> pages. We might want to reach out to those people to see if we should
>>> include those project already or just add them later on.
>>>
>>> Please give feedback.
>>>
>>>
>>> -Matthias
>>>
>>>
>>>
>>>
>>> On 10/09/2015 03:34 PM, Maximilian Michels wrote:
>>>> Yes, Community is a better place. You can also add the Dataflow Runner
>>>> https://github.com/dataArtisans/flink-dataflow.
>>>>
>>>> On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
>>>> <[hidden email]> wrote:
>>>>> Thank you Matthias!
>>>>>
>>>>> I'm not sure where the "Downloads" section is the right place for this.
>>>>> I would actually put it under "Community", with a header "External
>>>>> Contributions" or something like this, but I'm not feeling strong about
>>>>> this :)
>>>>>
>>>>> -Vasia.
>>>>>
>>>>>
>>>>> On 9 October 2015 at 15:29, Matthias J. Sax <[hidden email]> wrote:
>>>>>
>>>>>> I was not sure what we should add and was hoping for input from the
>>>>>> community.
>>>>>>
>>>>>> I am aware of the following projects we might want to add:
>>>>>>
>>>>>>   - Zeppelin
>>>>>>   - SAMOA
>>>>>>   - Mahout
>>>>>>   - Cascading (dataartisan repo)
>>>>>>   - BigPetStore
>>>>>>   - Gradoop
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>>>>>>> Cool. Right now the list is empty. Do you already have a list you
>>>>>>> could include in the upcoming pull request? :)
>>>>>>>
>>>>>>> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <[hidden email]>
>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I just started this. Please see
>>>>>>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>>>>>>
>>>>>>>> I think, it is the best way to extend the "Downloads" page. I would also
>>>>>>>> add a link to this on the main page's "Getting Started" section.
>>>>>>>>
>>>>>>>> As a first try, I started like this:
>>>>>>>>> Third party packages
>>>>>>>>>
>>>>>>>>> This is a list of third party packages (ie, libraries, system
>>>>>> extensions, or examples) build for Flink. The Flink community only collects
>>>>>> links to those packages but does not maintain them. Thus, they do not
>>>>>> belong to the Apache Flink project and the community cannot give any
>>>>>> support for them.
>>>>>>>>> Package Name
>>>>>>>>>
>>>>>>>>> Available for Flink 0.8.x and 0.9.x
>>>>>>>>>
>>>>>>>>> Short description
>>>>>>>>>
>>>>>>>>> Please let us know, if we missed to list your package. Be aware, that
>>>>>> we might remove listed packages without notice.
>>>>>>>>
>>>>>>>> Can you please give me some input, what projects I should add initially?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>>>>>>>> IMHO we can do that. There should be a disclaimer that the third party
>>>>>>>>> software is not officially supported.
>>>>>>>>>
>>>>>>>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <[hidden email]>
>>>>>> wrote:
>>>>>>>>>> Should we add a new page at Flink project web page?
>>>>>>>>>>
>>>>>>>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>>>>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>>>>>>>>> party software using Flink should be enough for now; of course,
>>>>>>>>>>> outside the Apache realm.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <[hidden email]>
>>>>>> wrote:
>>>>>>>>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
>>>>>> like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>>>>>>>
>>>>>>>>>>>> [1] http://spark-packages.org
>>>>>>>>>>>>
>>>>>>>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <[hidden email]>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>>>>>>>> release. From my point of view, there should be a single parent
>>>>>> module
>>>>>>>>>>>>> that contains *independent modules* for each extension/library
>>>>>> (there
>>>>>>>>>>>>> should be no "cross dependencies" between the modules and each
>>>>>> module
>>>>>>>>>>>>> can specify the flink dependencies it needs by itself). This make
>>>>>> is
>>>>>>>>>>>>> most flexible. And if a library works on an old release, it might
>>>>>> just
>>>>>>>>>>>>> stay there as is. If a library changes (due to Flink changes), it
>>>>>> might
>>>>>>>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Each module should provide a short doc (README) that shows how to
>>>>>> use an
>>>>>>>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>>>>>>>> contributor to maintain the library. If it breaks and is not
>>>>>> maintained
>>>>>>>>>>>>> any further, we can simple remove it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>>>>>>>> extension/libraries right now. I would put the responsibility
>>>>>> (more or
>>>>>>>>>>>>> less completely) on the contributor and delete project that do not
>>>>>> fix
>>>>>>>>>>>>> any more.
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Vasia: a link to a library could be included in the README. If
>>>>>> anybody
>>>>>>>>>>>>> only wants to share a library but not contribute code, the parent
>>>>>> README
>>>>>>>>>>>>> could contain a list of additional links.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>>>>>>>> How about, for now, we simply create a page where we gather
>>>>>> links/short
>>>>>>>>>>>>>> descriptions of all these contributions
>>>>>>>>>>>>>> and let the maintenance and dependency management to the
>>>>>> tool/library
>>>>>>>>>>>>>> creators?
>>>>>>>>>>>>>> This way we will at least have these contributions in one place
>>>>>> and link to
>>>>>>>>>>>>>> them somewhere from the website.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Vasia.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <[hidden email]>
>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for bringing up this idea. Actually, it has been
>>>>>> discussed a
>>>>>>>>>>>>>>> couple of times on the mailing list whether we should have a
>>>>>> central
>>>>>>>>>>>>>>> place for third-party extensions/contributions/libraries. This
>>>>>> could
>>>>>>>>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>>>>>>>>> repository.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In the long term, we really need a solution for these questions.
>>>>>> The
>>>>>>>>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>>>>>>>>> flink-contrib/flink-staging. I would find something
>>>>>> packaged-based
>>>>>>>>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>>>>>>>>> developers. At the current pace of Flink development, collecting
>>>>>> these
>>>>>>>>>>>>>>> contributions externally without properly maintaining them,
>>>>>> doesn't
>>>>>>>>>>>>>>> make much sense to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <
>>>>>> [hidden email]> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> many people are building quite exiting stuff on top of Flink.
>>>>>> It is hard
>>>>>>>>>>>>>>>> to keep an good overview on what stuff is available and what
>>>>>> not. What
>>>>>>>>>>>>>>>> do you think about starting a second git repository
>>>>>> "flink-external"
>>>>>>>>>>>>>>>> that collects all those code?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The ideas would be to collect stuff in a central point, such
>>>>>> that people
>>>>>>>>>>>>>>>> can access it easily and get an overview what is already
>>>>>> available (this
>>>>>>>>>>>>>>>> might also avoid duplicate development). It might also be a
>>>>>> good point
>>>>>>>>>>>>>>>> to show common patterns. In order to collect as much as
>>>>>> possible, the
>>>>>>>>>>>>>>>> contributing requirement (with respect to testing etc) could be
>>>>>> lower
>>>>>>>>>>>>>>>> than for Flink itself.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For example, I recently started a small flink-clojure module
>>>>>> with a
>>>>>>>>>>>>>>>> simple word-count example to answer a question on SO. Including
>>>>>> this in
>>>>>>>>>>>>>>>> Flink would not be appropriate. However, for a flink-external
>>>>>> repro it
>>>>>>>>>>>>>>>> might be nice to have.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Chiwan Park
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>
>>