(DEPRECATED) Apache Flink Mailing List archive.

Pulling Streaming out of staging and project restructure

Classic

List

Threaded

29 messages Options

Márton Balassi

Re: Pulling Streaming out of staging and project restructure

@Matthias: +1.

On Fri, Oct 2, 2015 at 11:27 AM, Stephan Ewen <[hidden email]> wrote:

> @matthias +1 for that approach
>
> On Fri, Oct 2, 2015 at 11:21 AM, Matthias J. Sax <[hidden email]> wrote:
>
> > It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> > would be the cleanest solution.
> >
> > So in flink-contrib there would be two modules:
> > - flink-storm
> > - flink-storm-examples
> >
> > Please let me know if you have any objection about it.
> >
> > -Matthias
> >
> > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > Sure. Will do that.
> > >
> > > -Matthias
> > >
> > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > >> @Matthias: How about getting rid of the storm-compatibility-parent and
> > >> making the core and examples projects directly projects in "contrib"
> > >>
> > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]>
> > wrote:
> > >>
> > >>> +1 for the new project structure. Getting rid of our code dump is a
> > good
> > >>> thing.
> > >>>
> > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]>
> > >>> wrote:
> > >>>
> > >>>> +1 Matthias, let's limit the overhead this has for the module
> > >>> maintainers.
> > >>>>
> > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]>
> > >>> wrote:
> > >>>>> I will commit something to flink-storm-compatibility tomorrow that
> > >>>>> contains some internal package restructuring. I think, renaming the
> > >>>>> three modules in this commit would be a smart move as both changes
> > >>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
> > >>> this
> > >>>>> pain to a single time. If no objections, I will commit those
> changes
> > >>>>> tomorrow.
> > >>>>>
> > >>>>> -Matthias
> > >>>>>
> > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > >>>>>> +1
> > >>>>>>
> > >>>>>> I like the idea moving "staging" projects into appropriate
> modules.
> > >>>>>>
> > >>>>>> While we are at it, I would like to propose changing "
> > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket
> > list
> > >>>>>> but would be nice if it is part of re-org.
> > >>>>>> Supporting Hadoop in the connector implicitly means compatibility
> > with
> > >>>> Hadoop.
> > >>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
> > >>>>>>
> > >>>>>> - Henry
> > >>>>>>
> > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]>
> > >>> wrote:
> > >>>>>>> Hi all!
> > >>>>>>>
> > >>>>>>> We are making good headway with reworking the last parts of the
> > >>> Window
> > >>>> API.
> > >>>>>>> After that, the streaming API should be good to be pulled out of
> > >>>> staging.
> > >>>>>>>
> > >>>>>>> Since we are reorganizing the projects as part of that, I would
> > shift
> > >>>> a bit
> > >>>>>>> more to bring things a bit more up to date.
> > >>>>>>>
> > >>>>>>> In this restructure, I would like to get rid of the
> "flink-staging"
> > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > difference
> > >>>>>>> whether a project is in "staging" or not, so it does not help
> much
> > to
> > >>>> have
> > >>>>>>> that directory structure.
> > >>>>>>> On the other hand, projects have a tendency to linger in staging
> > >>>> forever
> > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > >>>>>>>
> > >>>>>>> The new structure could be
> > >>>>>>>
> > >>>>>>> flink-core
> > >>>>>>> flink-java
> > >>>>>>> flink-scala
> > >>>>>>> flink-streaming-core
> > >>>>>>> flink-streaming-scala
> > >>>>>>>
> > >>>>>>> flink-runtime
> > >>>>>>> flink-runtime-web
> > >>>>>>> flink-optimizer
> > >>>>>>> flink-clients
> > >>>>>>>
> > >>>>>>> flink-shaded
> > >>>>>>> -> flink-shaded-hadoop
> > >>>>>>> -> flink-shaded-hadoop2
> > >>>>>>> -> flink-shaded-include-yarn-tests
> > >>>>>>> -> flink-shaded-curator
> > >>>>>>>
> > >>>>>>> flink-examples
> > >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming)
> > >>>>>>>
> > >>>>>>> flink-batch-connectors
> > >>>>>>> -> flink-avro
> > >>>>>>> -> flink-jdbc
> > >>>>>>> -> flink-hadoop-compatibility
> > >>>>>>> -> flink-hbase
> > >>>>>>> -> flink-hcatalog
> > >>>>>>>
> > >>>>>>> flink-streaming-connectors
> > >>>>>>> -> flink-connector-twitter
> > >>>>>>> -> flink-streaming-examples
> > >>>>>>> -> flink-connector-flume
> > >>>>>>> -> flink-connector-kafka
> > >>>>>>> -> flink-connector-elasticsearch
> > >>>>>>> -> flink-connector-rabbitmq
> > >>>>>>> -> flink-connector-filesystem
> > >>>>>>>
> > >>>>>>> flink-libraries
> > >>>>>>> -> flink-gelly
> > >>>>>>> -> flink-gelly-scala
> > >>>>>>> -> flink-ml
> > >>>>>>> -> flink-table
> > >>>>>>> -> flink-language-binding
> > >>>>>>> -> flink-python
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> flink-scala-shell
> > >>>>>>>
> > >>>>>>> flink-test-utils
> > >>>>>>> flink-tests
> > >>>>>>> flink-fs-tests
> > >>>>>>>
> > >>>>>>> flink-contrib
> > >>>>>>> -> flink-storm-compatibility
> > >>>>>>> -> flink-storm-compatibility-examples
> > >>>>>>> -> flink-streaming-utils
> > >>>>>>> -> flink-tweet-inputformat
> > >>>>>>> -> flink-operator-stats
> > >>>>>>> -> flink-tez
> > >>>>>>>
> > >>>>>>> flink-quickstart
> > >>>>>>> -> flink-quickstart-java
> > >>>>>>> -> flink-quickstart-scala
> > >>>>>>> -> flink-tez-quickstart
> > >>>>>>>
> > >>>>>>> flink-yarn
> > >>>>>>> flink-yarn-tests
> > >>>>>>>
> > >>>>>>> flink-dist
> > >>>>>>>
> > >>>>>>> flink-benchmark
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Let me know if that makes sense!
> > >>>>>>>
> > >>>>>>> Greetings,
> > >>>>>>> Stephan
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>

Aljoscha Krettek-2

Re: Pulling Streaming out of staging and project restructure

+1

On Fri, 2 Oct 2015 at 11:37 Márton Balassi <[hidden email]> wrote:

> @Matthias: +1.
>
> On Fri, Oct 2, 2015 at 11:27 AM, Stephan Ewen <[hidden email]> wrote:
>
> > @matthias +1 for that approach
> >
> > On Fri, Oct 2, 2015 at 11:21 AM, Matthias J. Sax <[hidden email]>
> wrote:
> >
> > > It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> > > would be the cleanest solution.
> > >
> > > So in flink-contrib there would be two modules:
> > > - flink-storm
> > > - flink-storm-examples
> > >
> > > Please let me know if you have any objection about it.
> > >
> > > -Matthias
> > >
> > > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > > Sure. Will do that.
> > > >
> > > > -Matthias
> > > >
> > > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > > >> @Matthias: How about getting rid of the storm-compatibility-parent
> and
> > > >> making the core and examples projects directly projects in "contrib"
> > > >>
> > > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <
> [hidden email]>
> > > wrote:
> > > >>
> > > >>> +1 for the new project structure. Getting rid of our code dump is a
> > > good
> > > >>> thing.
> > > >>>
> > > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <
> [hidden email]>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 Matthias, let's limit the overhead this has for the module
> > > >>> maintainers.
> > > >>>>
> > > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <
> [hidden email]>
> > > >>> wrote:
> > > >>>>> I will commit something to flink-storm-compatibility tomorrow
> that
> > > >>>>> contains some internal package restructuring. I think, renaming
> the
> > > >>>>> three modules in this commit would be a smart move as both
> changes
> > > >>>>> result in merge conflicts when rebasing open PRs. Thus we can
> limit
> > > >>> this
> > > >>>>> pain to a single time. If no objections, I will commit those
> > changes
> > > >>>>> tomorrow.
> > > >>>>>
> > > >>>>> -Matthias
> > > >>>>>
> > > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > > >>>>>> +1
> > > >>>>>>
> > > >>>>>> I like the idea moving "staging" projects into appropriate
> > modules.
> > > >>>>>>
> > > >>>>>> While we are at it, I would like to propose changing "
> > > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my
> bucket
> > > list
> > > >>>>>> but would be nice if it is part of re-org.
> > > >>>>>> Supporting Hadoop in the connector implicitly means
> compatibility
> > > with
> > > >>>> Hadoop.
> > > >>>>>> Also same thing with "flink-storm-compatibility" to
> "flink-storm".
> > > >>>>>>
> > > >>>>>> - Henry
> > > >>>>>>
> > > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]>
> > > >>> wrote:
> > > >>>>>>> Hi all!
> > > >>>>>>>
> > > >>>>>>> We are making good headway with reworking the last parts of the
> > > >>> Window
> > > >>>> API.
> > > >>>>>>> After that, the streaming API should be good to be pulled out
> of
> > > >>>> staging.
> > > >>>>>>>
> > > >>>>>>> Since we are reorganizing the projects as part of that, I would
> > > shift
> > > >>>> a bit
> > > >>>>>>> more to bring things a bit more up to date.
> > > >>>>>>>
> > > >>>>>>> In this restructure, I would like to get rid of the
> > "flink-staging"
> > > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > > difference
> > > >>>>>>> whether a project is in "staging" or not, so it does not help
> > much
> > > to
> > > >>>> have
> > > >>>>>>> that directory structure.
> > > >>>>>>> On the other hand, projects have a tendency to linger in
> staging
> > > >>>> forever
> > > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > > >>>>>>>
> > > >>>>>>> The new structure could be
> > > >>>>>>>
> > > >>>>>>> flink-core
> > > >>>>>>> flink-java
> > > >>>>>>> flink-scala
> > > >>>>>>> flink-streaming-core
> > > >>>>>>> flink-streaming-scala
> > > >>>>>>>
> > > >>>>>>> flink-runtime
> > > >>>>>>> flink-runtime-web
> > > >>>>>>> flink-optimizer
> > > >>>>>>> flink-clients
> > > >>>>>>>
> > > >>>>>>> flink-shaded
> > > >>>>>>> -> flink-shaded-hadoop
> > > >>>>>>> -> flink-shaded-hadoop2
> > > >>>>>>> -> flink-shaded-include-yarn-tests
> > > >>>>>>> -> flink-shaded-curator
> > > >>>>>>>
> > > >>>>>>> flink-examples
> > > >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming)
> > > >>>>>>>
> > > >>>>>>> flink-batch-connectors
> > > >>>>>>> -> flink-avro
> > > >>>>>>> -> flink-jdbc
> > > >>>>>>> -> flink-hadoop-compatibility
> > > >>>>>>> -> flink-hbase
> > > >>>>>>> -> flink-hcatalog
> > > >>>>>>>
> > > >>>>>>> flink-streaming-connectors
> > > >>>>>>> -> flink-connector-twitter
> > > >>>>>>> -> flink-streaming-examples
> > > >>>>>>> -> flink-connector-flume
> > > >>>>>>> -> flink-connector-kafka
> > > >>>>>>> -> flink-connector-elasticsearch
> > > >>>>>>> -> flink-connector-rabbitmq
> > > >>>>>>> -> flink-connector-filesystem
> > > >>>>>>>
> > > >>>>>>> flink-libraries
> > > >>>>>>> -> flink-gelly
> > > >>>>>>> -> flink-gelly-scala
> > > >>>>>>> -> flink-ml
> > > >>>>>>> -> flink-table
> > > >>>>>>> -> flink-language-binding
> > > >>>>>>> -> flink-python
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> flink-scala-shell
> > > >>>>>>>
> > > >>>>>>> flink-test-utils
> > > >>>>>>> flink-tests
> > > >>>>>>> flink-fs-tests
> > > >>>>>>>
> > > >>>>>>> flink-contrib
> > > >>>>>>> -> flink-storm-compatibility
> > > >>>>>>> -> flink-storm-compatibility-examples
> > > >>>>>>> -> flink-streaming-utils
> > > >>>>>>> -> flink-tweet-inputformat
> > > >>>>>>> -> flink-operator-stats
> > > >>>>>>> -> flink-tez
> > > >>>>>>>
> > > >>>>>>> flink-quickstart
> > > >>>>>>> -> flink-quickstart-java
> > > >>>>>>> -> flink-quickstart-scala
> > > >>>>>>> -> flink-tez-quickstart
> > > >>>>>>>
> > > >>>>>>> flink-yarn
> > > >>>>>>> flink-yarn-tests
> > > >>>>>>>
> > > >>>>>>> flink-dist
> > > >>>>>>>
> > > >>>>>>> flink-benchmark
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Let me know if that makes sense!
> > > >>>>>>>
> > > >>>>>>> Greetings,
> > > >>>>>>> Stephan
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

Henry Saputra

Re: Pulling Streaming out of staging and project restructure

In reply to this post by Matthias J. Sax-2

+1

On Friday, October 2, 2015, Matthias J. Sax <[hidden email]> wrote:

> It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> would be the cleanest solution.
>
> So in flink-contrib there would be two modules:
> - flink-storm
> - flink-storm-examples
>
> Please let me know if you have any objection about it.
>
> -Matthias
>
> On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > Sure. Will do that.
> >
> > -Matthias
> >
> > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> >> @Matthias: How about getting rid of the storm-compatibility-parent and
> >> making the core and examples projects directly projects in "contrib"
> >>
> >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]
> <javascript:;>> wrote:
> >>
> >>> +1 for the new project structure. Getting rid of our code dump is a
> good
> >>> thing.
> >>>
> >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]
> <javascript:;>>
> >>> wrote:
> >>>
> >>>> +1 Matthias, let's limit the overhead this has for the module
> >>> maintainers.
> >>>>
> >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]
> <javascript:;>>
> >>> wrote:
> >>>>> I will commit something to flink-storm-compatibility tomorrow that
> >>>>> contains some internal package restructuring. I think, renaming the
> >>>>> three modules in this commit would be a smart move as both changes
> >>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
> >>> this
> >>>>> pain to a single time. If no objections, I will commit those changes
> >>>>> tomorrow.
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> >>>>>> +1
> >>>>>>
> >>>>>> I like the idea moving "staging" projects into appropriate modules.
> >>>>>>
> >>>>>> While we are at it, I would like to propose changing "
> >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket
> list
> >>>>>> but would be nice if it is part of re-org.
> >>>>>> Supporting Hadoop in the connector implicitly means compatibility
> with
> >>>> Hadoop.
> >>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
> >>>>>>
> >>>>>> - Henry
> >>>>>>
> >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]
> <javascript:;>>
> >>> wrote:
> >>>>>>> Hi all!
> >>>>>>>
> >>>>>>> We are making good headway with reworking the last parts of the
> >>> Window
> >>>> API.
> >>>>>>> After that, the streaming API should be good to be pulled out of
> >>>> staging.
> >>>>>>>
> >>>>>>> Since we are reorganizing the projects as part of that, I would
> shift
> >>>> a bit
> >>>>>>> more to bring things a bit more up to date.
> >>>>>>>
> >>>>>>> In this restructure, I would like to get rid of the "flink-staging"
> >>>>>>> project. Anyone who only uses the maven artifacts sees no
> difference
> >>>>>>> whether a project is in "staging" or not, so it does not help much
> to
> >>>> have
> >>>>>>> that directory structure.
> >>>>>>> On the other hand, projects have a tendency to linger in staging
> >>>> forever
> >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> >>>>>>>
> >>>>>>> The new structure could be
> >>>>>>>
> >>>>>>> flink-core
> >>>>>>> flink-java
> >>>>>>> flink-scala
> >>>>>>> flink-streaming-core
> >>>>>>> flink-streaming-scala
> >>>>>>>
> >>>>>>> flink-runtime
> >>>>>>> flink-runtime-web
> >>>>>>> flink-optimizer
> >>>>>>> flink-clients
> >>>>>>>
> >>>>>>> flink-shaded
> >>>>>>> -> flink-shaded-hadoop
> >>>>>>> -> flink-shaded-hadoop2
> >>>>>>> -> flink-shaded-include-yarn-tests
> >>>>>>> -> flink-shaded-curator
> >>>>>>>
> >>>>>>> flink-examples
> >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming)
> >>>>>>>
> >>>>>>> flink-batch-connectors
> >>>>>>> -> flink-avro
> >>>>>>> -> flink-jdbc
> >>>>>>> -> flink-hadoop-compatibility
> >>>>>>> -> flink-hbase
> >>>>>>> -> flink-hcatalog
> >>>>>>>
> >>>>>>> flink-streaming-connectors
> >>>>>>> -> flink-connector-twitter
> >>>>>>> -> flink-streaming-examples
> >>>>>>> -> flink-connector-flume
> >>>>>>> -> flink-connector-kafka
> >>>>>>> -> flink-connector-elasticsearch
> >>>>>>> -> flink-connector-rabbitmq
> >>>>>>> -> flink-connector-filesystem
> >>>>>>>
> >>>>>>> flink-libraries
> >>>>>>> -> flink-gelly
> >>>>>>> -> flink-gelly-scala
> >>>>>>> -> flink-ml
> >>>>>>> -> flink-table
> >>>>>>> -> flink-language-binding
> >>>>>>> -> flink-python
> >>>>>>>
> >>>>>>>
> >>>>>>> flink-scala-shell
> >>>>>>>
> >>>>>>> flink-test-utils
> >>>>>>> flink-tests
> >>>>>>> flink-fs-tests
> >>>>>>>
> >>>>>>> flink-contrib
> >>>>>>> -> flink-storm-compatibility
> >>>>>>> -> flink-storm-compatibility-examples
> >>>>>>> -> flink-streaming-utils
> >>>>>>> -> flink-tweet-inputformat
> >>>>>>> -> flink-operator-stats
> >>>>>>> -> flink-tez
> >>>>>>>
> >>>>>>> flink-quickstart
> >>>>>>> -> flink-quickstart-java
> >>>>>>> -> flink-quickstart-scala
> >>>>>>> -> flink-tez-quickstart
> >>>>>>>
> >>>>>>> flink-yarn
> >>>>>>> flink-yarn-tests
> >>>>>>>
> >>>>>>> flink-dist
> >>>>>>>
> >>>>>>> flink-benchmark
> >>>>>>>
> >>>>>>>
> >>>>>>> Let me know if that makes sense!
> >>>>>>>
> >>>>>>> Greetings,
> >>>>>>> Stephan
> >>>>>
> >>>>
> >>>
> >>
> >
>
>

Fabian Hueske-2

RE: Pulling Streaming out of staging and project restructure

+1

From: Henry Saputra
Sent: Friday, October 2, 2015 19:34
To: [hidden email]
Subject: Re: Pulling Streaming out of staging and project restructure

+1

On Friday, October 2, 2015, Matthias J. Sax <[hidden email]> wrote:

Robert Metzger

Re: Pulling Streaming out of staging and project restructure

The renaming hasn't been done completely. I would like to fix this before
the 1.0.0 release.

I think the following issues are still open:
- Merge flink-java-examples, flink-scala-examples and
flink-streaming-examples into "flink-examples" + build the streaming
examples for the binary build as well.
- move flink-table to flink-libraries
- move some modules from flink-staging to flink-batch-connectors

If nobody disagrees, I'll open a PR for these changes.
I created a JIRA for this discussion:
https://issues.apache.org/jira/browse/FLINK-3205

On Fri, Oct 2, 2015 at 7:58 PM, fhueske <[hidden email]> wrote:

> +1
>
>
> From: Henry Saputra
> Sent: Friday, October 2, 2015 19:34
> To: [hidden email]
> Subject: Re: Pulling Streaming out of staging and project restructure
>
>
> +1
>
> On Friday, October 2, 2015, Matthias J. Sax <[hidden email]> wrote:
>
> > It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> > would be the cleanest solution.
> >
> > So in flink-contrib there would be two modules:
> > - flink-storm
> > - flink-storm-examples
> >
> > Please let me know if you have any objection about it.
> >
> > -Matthias
> >
> > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > Sure. Will do that.
> > >
> > > -Matthias
> > >
> > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > >> @Matthias: How about getting rid of the storm-compatibility-parent and
> > >> making the core and examples projects directly projects in "contrib"
> > >>
> > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]
> > <javascript:;>> wrote:
> > >>
> > >>> +1 for the new project structure. Getting rid of our code dump is a
> > good
> > >>> thing.
> > >>>
> > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]
> > <javascript:;>>
> > >>> wrote:
> > >>>
> > >>>> +1 Matthias, let's limit the overhead this has for the module
> > >>> maintainers.
> > >>>>
> > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]
> > <javascript:;>>
> > >>> wrote:
> > >>>>> I will commit something to flink-storm-compatibility tomorrow that
> > >>>>> contains some internal package restructuring. I think, renaming the
> > >>>>> three modules in this commit would be a smart move as both changes
> > >>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
> > >>> this
> > >>>>> pain to a single time. If no objections, I will commit those
> changes
> > >>>>> tomorrow.
> > >>>>>
> > >>>>> -Matthias
> > >>>>>
> > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > >>>>>> +1
> > >>>>>>
> > >>>>>> I like the idea moving "staging" projects into appropriate
> modules.
> > >>>>>>
> > >>>>>> While we are at it, I would like to propose changing "
> > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket
> > list
> > >>>>>> but would be nice if it is part of re-org.
> > >>>>>> Supporting Hadoop in the connector implicitly means compatibility
> > with
> > >>>> Hadoop.
> > >>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
> > >>>>>>
> > >>>>>> - Henry
> > >>>>>>
> > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]
> > <javascript:;>>
> > >>> wrote:
> > >>>>>>> Hi all!
> > >>>>>>>
> > >>>>>>> We are making good headway with reworking the last parts of the
> > >>> Window
> > >>>> API.
> > >>>>>>> After that, the streaming API should be good to be pulled out of
> > >>>> staging.
> > >>>>>>>
> > >>>>>>> Since we are reorganizing the projects as part of that, I would
> > shift
> > >>>> a bit
> > >>>>>>> more to bring things a bit more up to date.
> > >>>>>>>
> > >>>>>>> In this restructure, I would like to get rid of the
> "flink-staging"
> > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > difference
> > >>>>>>> whether a project is in "staging" or not, so it does not help
> much
> > to
> > >>>> have
> > >>>>>>> that directory structure.
> > >>>>>>> On the other hand, projects have a tendency to linger in staging
> > >>>> forever
> > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > >>>>>>>
> > >>>>>>> The new structure could be
> > >>>>>>>
> > >>>>>>> flink-core
> > >>>>>>> flink-java
> > >>>>>>> flink-scala
> > >>>>>>> flink-streaming-core
> > >>>>>>> flink-streaming-scala
> > >>>>>>>
> > >>>>>>> flink-runtime
> > >>>>>>> flink-runtime-web
> > >>>>>>> flink-optimizer
> > >>>>>>> flink-clients
> > >>>>>>>
> > >>>>>>> flink-shaded
> > >>>>>>> -> flink-shaded-hadoop
> > >>>>>>> -> flink-shaded-hadoop2
> > >>>>>>> -> flink-shaded-include-yarn-tests
> > >>>>>>> -> flink-shaded-curator
> > >>>>>>>
> > >>>>>>> flink-examples
> > >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming)
> > >>>>>>>
> > >>>>>>> flink-batch-connectors
> > >>>>>>> -> flink-avro
> > >>>>>>> -> flink-jdbc
> > >>>>>>> -> flink-hadoop-compatibility
> > >>>>>>> -> flink-hbase
> > >>>>>>> -> flink-hcatalog
> > >>>>>>>
> > >>>>>>> flink-streaming-connectors
> > >>>>>>> -> flink-connector-twitter
> > >>>>>>> -> flink-streaming-examples
> > >>>>>>> -> flink-connector-flume
> > >>>>>>> -> flink-connector-kafka
> > >>>>>>> -> flink-connector-elasticsearch
> > >>>>>>> -> flink-connector-rabbitmq
> > >>>>>>> -> flink-connector-filesystem
> > >>>>>>>
> > >>>>>>> flink-libraries
> > >>>>>>> -> flink-gelly
> > >>>>>>> -> flink-gelly-scala
> > >>>>>>> -> flink-ml
> > >>>>>>> -> flink-table
> > >>>>>>> -> flink-language-binding
> > >>>>>>> -> flink-python
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> flink-scala-shell
> > >>>>>>>
> > >>>>>>> flink-test-utils
> > >>>>>>> flink-tests
> > >>>>>>> flink-fs-tests
> > >>>>>>>
> > >>>>>>> flink-contrib
> > >>>>>>> -> flink-storm-compatibility
> > >>>>>>> -> flink-storm-compatibility-examples
> > >>>>>>> -> flink-streaming-utils
> > >>>>>>> -> flink-tweet-inputformat
> > >>>>>>> -> flink-operator-stats
> > >>>>>>> -> flink-tez
> > >>>>>>>
> > >>>>>>> flink-quickstart
> > >>>>>>> -> flink-quickstart-java
> > >>>>>>> -> flink-quickstart-scala
> > >>>>>>> -> flink-tez-quickstart
> > >>>>>>>
> > >>>>>>> flink-yarn
> > >>>>>>> flink-yarn-tests
> > >>>>>>>
> > >>>>>>> flink-dist
> > >>>>>>>
> > >>>>>>> flink-benchmark
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Let me know if that makes sense!
> > >>>>>>>
> > >>>>>>> Greetings,
> > >>>>>>> Stephan
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>
>
>

Fabian Hueske-2

Re: Pulling Streaming out of staging and project restructure

Stephan already opened a PR for restructuring the examples:

--> https://github.com/apache/flink/pull/1482

Otherwise +1!

2016-01-06 16:21 GMT+01:00 Robert Metzger <[hidden email]>:

> The renaming hasn't been done completely. I would like to fix this before
> the 1.0.0 release.
>
> I think the following issues are still open:
> - Merge flink-java-examples, flink-scala-examples and
> flink-streaming-examples into "flink-examples" + build the streaming
> examples for the binary build as well.
> - move flink-table to flink-libraries
> - move some modules from flink-staging to flink-batch-connectors
>
>
> If nobody disagrees, I'll open a PR for these changes.
> I created a JIRA for this discussion:
> https://issues.apache.org/jira/browse/FLINK-3205
>
>
> On Fri, Oct 2, 2015 at 7:58 PM, fhueske <[hidden email]> wrote:
>
> > +1
> >
> >
> > From: Henry Saputra
> > Sent: Friday, October 2, 2015 19:34
> > To: [hidden email]
> > Subject: Re: Pulling Streaming out of staging and project restructure
> >
> >
> > +1
> >
> > On Friday, October 2, 2015, Matthias J. Sax <[hidden email]> wrote:
> >
> > > It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> > > would be the cleanest solution.
> > >
> > > So in flink-contrib there would be two modules:
> > > - flink-storm
> > > - flink-storm-examples
> > >
> > > Please let me know if you have any objection about it.
> > >
> > > -Matthias
> > >
> > > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > > Sure. Will do that.
> > > >
> > > > -Matthias
> > > >
> > > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > > >> @Matthias: How about getting rid of the storm-compatibility-parent
> and
> > > >> making the core and examples projects directly projects in "contrib"
> > > >>
> > > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <
> [hidden email]
> > > <javascript:;>> wrote:
> > > >>
> > > >>> +1 for the new project structure. Getting rid of our code dump is a
> > > good
> > > >>> thing.
> > > >>>
> > > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <
> [hidden email]
> > > <javascript:;>>
> > > >>> wrote:
> > > >>>
> > > >>>> +1 Matthias, let's limit the overhead this has for the module
> > > >>> maintainers.
> > > >>>>
> > > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <
> [hidden email]
> > > <javascript:;>>
> > > >>> wrote:
> > > >>>>> I will commit something to flink-storm-compatibility tomorrow
> that
> > > >>>>> contains some internal package restructuring. I think, renaming
> the
> > > >>>>> three modules in this commit would be a smart move as both
> changes
> > > >>>>> result in merge conflicts when rebasing open PRs. Thus we can
> limit
> > > >>> this
> > > >>>>> pain to a single time. If no objections, I will commit those
> > changes
> > > >>>>> tomorrow.
> > > >>>>>
> > > >>>>> -Matthias
> > > >>>>>
> > > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > > >>>>>> +1
> > > >>>>>>
> > > >>>>>> I like the idea moving "staging" projects into appropriate
> > modules.
> > > >>>>>>
> > > >>>>>> While we are at it, I would like to propose changing "
> > > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my
> bucket
> > > list
> > > >>>>>> but would be nice if it is part of re-org.
> > > >>>>>> Supporting Hadoop in the connector implicitly means
> compatibility
> > > with
> > > >>>> Hadoop.
> > > >>>>>> Also same thing with "flink-storm-compatibility" to
> "flink-storm".
> > > >>>>>>
> > > >>>>>> - Henry
> > > >>>>>>
> > > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]
> > > <javascript:;>>
> > > >>> wrote:
> > > >>>>>>> Hi all!
> > > >>>>>>>
> > > >>>>>>> We are making good headway with reworking the last parts of the
> > > >>> Window
> > > >>>> API.
> > > >>>>>>> After that, the streaming API should be good to be pulled out
> of
> > > >>>> staging.
> > > >>>>>>>
> > > >>>>>>> Since we are reorganizing the projects as part of that, I would
> > > shift
> > > >>>> a bit
> > > >>>>>>> more to bring things a bit more up to date.
> > > >>>>>>>
> > > >>>>>>> In this restructure, I would like to get rid of the
> > "flink-staging"
> > > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > > difference
> > > >>>>>>> whether a project is in "staging" or not, so it does not help
> > much
> > > to
> > > >>>> have
> > > >>>>>>> that directory structure.
> > > >>>>>>> On the other hand, projects have a tendency to linger in
> staging
> > > >>>> forever
> > > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > > >>>>>>>
> > > >>>>>>> The new structure could be
> > > >>>>>>>
> > > >>>>>>> flink-core
> > > >>>>>>> flink-java
> > > >>>>>>> flink-scala
> > > >>>>>>> flink-streaming-core
> > > >>>>>>> flink-streaming-scala
> > > >>>>>>>
> > > >>>>>>> flink-runtime
> > > >>>>>>> flink-runtime-web
> > > >>>>>>> flink-optimizer
> > > >>>>>>> flink-clients
> > > >>>>>>>
> > > >>>>>>> flink-shaded
> > > >>>>>>> -> flink-shaded-hadoop
> > > >>>>>>> -> flink-shaded-hadoop2
> > > >>>>>>> -> flink-shaded-include-yarn-tests
> > > >>>>>>> -> flink-shaded-curator
> > > >>>>>>>
> > > >>>>>>> flink-examples
> > > >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming)
> > > >>>>>>>
> > > >>>>>>> flink-batch-connectors
> > > >>>>>>> -> flink-avro
> > > >>>>>>> -> flink-jdbc
> > > >>>>>>> -> flink-hadoop-compatibility
> > > >>>>>>> -> flink-hbase
> > > >>>>>>> -> flink-hcatalog
> > > >>>>>>>
> > > >>>>>>> flink-streaming-connectors
> > > >>>>>>> -> flink-connector-twitter
> > > >>>>>>> -> flink-streaming-examples
> > > >>>>>>> -> flink-connector-flume
> > > >>>>>>> -> flink-connector-kafka
> > > >>>>>>> -> flink-connector-elasticsearch
> > > >>>>>>> -> flink-connector-rabbitmq
> > > >>>>>>> -> flink-connector-filesystem
> > > >>>>>>>
> > > >>>>>>> flink-libraries
> > > >>>>>>> -> flink-gelly
> > > >>>>>>> -> flink-gelly-scala
> > > >>>>>>> -> flink-ml
> > > >>>>>>> -> flink-table
> > > >>>>>>> -> flink-language-binding
> > > >>>>>>> -> flink-python
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> flink-scala-shell
> > > >>>>>>>
> > > >>>>>>> flink-test-utils
> > > >>>>>>> flink-tests
> > > >>>>>>> flink-fs-tests
> > > >>>>>>>
> > > >>>>>>> flink-contrib
> > > >>>>>>> -> flink-storm-compatibility
> > > >>>>>>> -> flink-storm-compatibility-examples
> > > >>>>>>> -> flink-streaming-utils
> > > >>>>>>> -> flink-tweet-inputformat
> > > >>>>>>> -> flink-operator-stats
> > > >>>>>>> -> flink-tez
> > > >>>>>>>
> > > >>>>>>> flink-quickstart
> > > >>>>>>> -> flink-quickstart-java
> > > >>>>>>> -> flink-quickstart-scala
> > > >>>>>>> -> flink-tez-quickstart
> > > >>>>>>>
> > > >>>>>>> flink-yarn
> > > >>>>>>> flink-yarn-tests
> > > >>>>>>>
> > > >>>>>>> flink-dist
> > > >>>>>>>
> > > >>>>>>> flink-benchmark
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Let me know if that makes sense!
> > > >>>>>>>
> > > >>>>>>> Greetings,
> > > >>>>>>> Stephan
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
> >
> >
>

Flavio Pompermaier

Re: Pulling Streaming out of staging and project restructure

I hope that it's not too late to suggest to add to restructuring also
https://issues.apache.org/jira/browse/FLINK-1827 so that to be able to
compile Flink without compiling also tests (-Dmaven.test.skip=true) and
save a lot of time...
I should be fairly easy to fix that.

Best,
Flavio

On Wed, Jan 6, 2016 at 4:22 PM, Fabian Hueske <[hidden email]> wrote:

> Stephan already opened a PR for restructuring the examples:
>
> --> https://github.com/apache/flink/pull/1482
>
> Otherwise +1!
>
> 2016-01-06 16:21 GMT+01:00 Robert Metzger <[hidden email]>:
>
> > The renaming hasn't been done completely. I would like to fix this before
> > the 1.0.0 release.
> >
> > I think the following issues are still open:
> > - Merge flink-java-examples, flink-scala-examples and
> > flink-streaming-examples into "flink-examples" + build the streaming
> > examples for the binary build as well.
> > - move flink-table to flink-libraries
> > - move some modules from flink-staging to flink-batch-connectors
> >
> >
> > If nobody disagrees, I'll open a PR for these changes.
> > I created a JIRA for this discussion:
> > https://issues.apache.org/jira/browse/FLINK-3205
> >
> >
> > On Fri, Oct 2, 2015 at 7:58 PM, fhueske <[hidden email]> wrote:
> >
> > > +1
> > >
> > >
> > > From: Henry Saputra
> > > Sent: Friday, October 2, 2015 19:34
> > > To: [hidden email]
> > > Subject: Re: Pulling Streaming out of staging and project restructure
> > >
> > >
> > > +1
> > >
> > > On Friday, October 2, 2015, Matthias J. Sax <[hidden email]> wrote:
> > >
> > > > It think, rename "flink-storm-compatibility-core" to just
> "flink-storm"
> > > > would be the cleanest solution.
> > > >
> > > > So in flink-contrib there would be two modules:
> > > > - flink-storm
> > > > - flink-storm-examples
> > > >
> > > > Please let me know if you have any objection about it.
> > > >
> > > > -Matthias
> > > >
> > > > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > > > Sure. Will do that.
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > > > >> @Matthias: How about getting rid of the storm-compatibility-parent
> > and
> > > > >> making the core and examples projects directly projects in
> "contrib"
> > > > >>
> > > > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <
> > [hidden email]
> > > > <javascript:;>> wrote:
> > > > >>
> > > > >>> +1 for the new project structure. Getting rid of our code dump
> is a
> > > > good
> > > > >>> thing.
> > > > >>>
> > > > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <
> > [hidden email]
> > > > <javascript:;>>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> +1 Matthias, let's limit the overhead this has for the module
> > > > >>> maintainers.
> > > > >>>>
> > > > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <
> > [hidden email]
> > > > <javascript:;>>
> > > > >>> wrote:
> > > > >>>>> I will commit something to flink-storm-compatibility tomorrow
> > that
> > > > >>>>> contains some internal package restructuring. I think, renaming
> > the
> > > > >>>>> three modules in this commit would be a smart move as both
> > changes
> > > > >>>>> result in merge conflicts when rebasing open PRs. Thus we can
> > limit
> > > > >>> this
> > > > >>>>> pain to a single time. If no objections, I will commit those
> > > changes
> > > > >>>>> tomorrow.
> > > > >>>>>
> > > > >>>>> -Matthias
> > > > >>>>>
> > > > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > > > >>>>>> +1
> > > > >>>>>>
> > > > >>>>>> I like the idea moving "staging" projects into appropriate
> > > modules.
> > > > >>>>>>
> > > > >>>>>> While we are at it, I would like to propose changing "
> > > > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my
> > bucket
> > > > list
> > > > >>>>>> but would be nice if it is part of re-org.
> > > > >>>>>> Supporting Hadoop in the connector implicitly means
> > compatibility
> > > > with
> > > > >>>> Hadoop.
> > > > >>>>>> Also same thing with "flink-storm-compatibility" to
> > "flink-storm".
> > > > >>>>>>
> > > > >>>>>> - Henry
> > > > >>>>>>
> > > > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <
> [hidden email]
> > > > <javascript:;>>
> > > > >>> wrote:
> > > > >>>>>>> Hi all!
> > > > >>>>>>>
> > > > >>>>>>> We are making good headway with reworking the last parts of
> the
> > > > >>> Window
> > > > >>>> API.
> > > > >>>>>>> After that, the streaming API should be good to be pulled out
> > of
> > > > >>>> staging.
> > > > >>>>>>>
> > > > >>>>>>> Since we are reorganizing the projects as part of that, I
> would
> > > > shift
> > > > >>>> a bit
> > > > >>>>>>> more to bring things a bit more up to date.
> > > > >>>>>>>
> > > > >>>>>>> In this restructure, I would like to get rid of the
> > > "flink-staging"
> > > > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > > > difference
> > > > >>>>>>> whether a project is in "staging" or not, so it does not help
> > > much
> > > > to
> > > > >>>> have
> > > > >>>>>>> that directory structure.
> > > > >>>>>>> On the other hand, projects have a tendency to linger in
> > staging
> > > > >>>> forever
> > > > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > > > >>>>>>>
> > > > >>>>>>> The new structure could be
> > > > >>>>>>>
> > > > >>>>>>> flink-core
> > > > >>>>>>> flink-java
> > > > >>>>>>> flink-scala
> > > > >>>>>>> flink-streaming-core
> > > > >>>>>>> flink-streaming-scala
> > > > >>>>>>>
> > > > >>>>>>> flink-runtime
> > > > >>>>>>> flink-runtime-web
> > > > >>>>>>> flink-optimizer
> > > > >>>>>>> flink-clients
> > > > >>>>>>>
> > > > >>>>>>> flink-shaded
> > > > >>>>>>> -> flink-shaded-hadoop
> > > > >>>>>>> -> flink-shaded-hadoop2
> > > > >>>>>>> -> flink-shaded-include-yarn-tests
> > > > >>>>>>> -> flink-shaded-curator
> > > > >>>>>>>
> > > > >>>>>>> flink-examples
> > > > >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming)
> > > > >>>>>>>
> > > > >>>>>>> flink-batch-connectors
> > > > >>>>>>> -> flink-avro
> > > > >>>>>>> -> flink-jdbc
> > > > >>>>>>> -> flink-hadoop-compatibility
> > > > >>>>>>> -> flink-hbase
> > > > >>>>>>> -> flink-hcatalog
> > > > >>>>>>>
> > > > >>>>>>> flink-streaming-connectors
> > > > >>>>>>> -> flink-connector-twitter
> > > > >>>>>>> -> flink-streaming-examples
> > > > >>>>>>> -> flink-connector-flume
> > > > >>>>>>> -> flink-connector-kafka
> > > > >>>>>>> -> flink-connector-elasticsearch
> > > > >>>>>>> -> flink-connector-rabbitmq
> > > > >>>>>>> -> flink-connector-filesystem
> > > > >>>>>>>
> > > > >>>>>>> flink-libraries
> > > > >>>>>>> -> flink-gelly
> > > > >>>>>>> -> flink-gelly-scala
> > > > >>>>>>> -> flink-ml
> > > > >>>>>>> -> flink-table
> > > > >>>>>>> -> flink-language-binding
> > > > >>>>>>> -> flink-python
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> flink-scala-shell
> > > > >>>>>>>
> > > > >>>>>>> flink-test-utils
> > > > >>>>>>> flink-tests
> > > > >>>>>>> flink-fs-tests
> > > > >>>>>>>
> > > > >>>>>>> flink-contrib
> > > > >>>>>>> -> flink-storm-compatibility
> > > > >>>>>>> -> flink-storm-compatibility-examples
> > > > >>>>>>> -> flink-streaming-utils
> > > > >>>>>>> -> flink-tweet-inputformat
> > > > >>>>>>> -> flink-operator-stats
> > > > >>>>>>> -> flink-tez
> > > > >>>>>>>
> > > > >>>>>>> flink-quickstart
> > > > >>>>>>> -> flink-quickstart-java
> > > > >>>>>>> -> flink-quickstart-scala
> > > > >>>>>>> -> flink-tez-quickstart
> > > > >>>>>>>
> > > > >>>>>>> flink-yarn
> > > > >>>>>>> flink-yarn-tests
> > > > >>>>>>>
> > > > >>>>>>> flink-dist
> > > > >>>>>>>
> > > > >>>>>>> flink-benchmark
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Let me know if that makes sense!
> > > > >>>>>>>
> > > > >>>>>>> Greetings,
> > > > >>>>>>> Stephan
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>

Stephan Ewen

Re: Pulling Streaming out of staging and project restructure

Hi Flavio!

You can always do "mvn -DskipTests clean package". That compiles tests, but
does not execute them.

Stephan

On Mon, Jan 11, 2016 at 11:34 AM, Flavio Pompermaier <[hidden email]>
wrote:

> I hope that it's not too late to suggest to add to restructuring also
> https://issues.apache.org/jira/browse/FLINK-1827 so that to be able to
> compile Flink without compiling also tests (-Dmaven.test.skip=true) and
> save a lot of time...
> I should be fairly easy to fix that.
>
> Best,
> Flavio
>
> On Wed, Jan 6, 2016 at 4:22 PM, Fabian Hueske <[hidden email]> wrote:
>
> > Stephan already opened a PR for restructuring the examples:
> >
> > --> https://github.com/apache/flink/pull/1482
> >
> > Otherwise +1!
> >
> > 2016-01-06 16:21 GMT+01:00 Robert Metzger <[hidden email]>:
> >
> > > The renaming hasn't been done completely. I would like to fix this
> before
> > > the 1.0.0 release.
> > >
> > > I think the following issues are still open:
> > > - Merge flink-java-examples, flink-scala-examples and
> > > flink-streaming-examples into "flink-examples" + build the streaming
> > > examples for the binary build as well.
> > > - move flink-table to flink-libraries
> > > - move some modules from flink-staging to flink-batch-connectors
> > >
> > >
> > > If nobody disagrees, I'll open a PR for these changes.
> > > I created a JIRA for this discussion:
> > > https://issues.apache.org/jira/browse/FLINK-3205
> > >
> > >
> > > On Fri, Oct 2, 2015 at 7:58 PM, fhueske <[hidden email]> wrote:
> > >
> > > > +1
> > > >
> > > >
> > > > From: Henry Saputra
> > > > Sent: Friday, October 2, 2015 19:34
> > > > To: [hidden email]
> > > > Subject: Re: Pulling Streaming out of staging and project restructure
> > > >
> > > >
> > > > +1
> > > >
> > > > On Friday, October 2, 2015, Matthias J. Sax <[hidden email]>
> wrote:
> > > >
> > > > > It think, rename "flink-storm-compatibility-core" to just
> > "flink-storm"
> > > > > would be the cleanest solution.
> > > > >
> > > > > So in flink-contrib there would be two modules:
> > > > > - flink-storm
> > > > > - flink-storm-examples
> > > > >
> > > > > Please let me know if you have any objection about it.
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > > > > Sure. Will do that.
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > > > > >> @Matthias: How about getting rid of the
> storm-compatibility-parent
> > > and
> > > > > >> making the core and examples projects directly projects in
> > "contrib"
> > > > > >>
> > > > > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <
> > > [hidden email]
> > > > > <javascript:;>> wrote:
> > > > > >>
> > > > > >>> +1 for the new project structure. Getting rid of our code dump
> > is a
> > > > > good
> > > > > >>> thing.
> > > > > >>>
> > > > > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <
> > > [hidden email]
> > > > > <javascript:;>>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> +1 Matthias, let's limit the overhead this has for the module
> > > > > >>> maintainers.
> > > > > >>>>
> > > > > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <
> > > [hidden email]
> > > > > <javascript:;>>
> > > > > >>> wrote:
> > > > > >>>>> I will commit something to flink-storm-compatibility tomorrow
> > > that
> > > > > >>>>> contains some internal package restructuring. I think,
> renaming
> > > the
> > > > > >>>>> three modules in this commit would be a smart move as both
> > > changes
> > > > > >>>>> result in merge conflicts when rebasing open PRs. Thus we can
> > > limit
> > > > > >>> this
> > > > > >>>>> pain to a single time. If no objections, I will commit those
> > > > changes
> > > > > >>>>> tomorrow.
> > > > > >>>>>
> > > > > >>>>> -Matthias
> > > > > >>>>>
> > > > > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > > > > >>>>>> +1
> > > > > >>>>>>
> > > > > >>>>>> I like the idea moving "staging" projects into appropriate
> > > > modules.
> > > > > >>>>>>
> > > > > >>>>>> While we are at it, I would like to propose changing "
> > > > > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my
> > > bucket
> > > > > list
> > > > > >>>>>> but would be nice if it is part of re-org.
> > > > > >>>>>> Supporting Hadoop in the connector implicitly means
> > > compatibility
> > > > > with
> > > > > >>>> Hadoop.
> > > > > >>>>>> Also same thing with "flink-storm-compatibility" to
> > > "flink-storm".
> > > > > >>>>>>
> > > > > >>>>>> - Henry
> > > > > >>>>>>
> > > > > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <
> > [hidden email]
> > > > > <javascript:;>>
> > > > > >>> wrote:
> > > > > >>>>>>> Hi all!
> > > > > >>>>>>>
> > > > > >>>>>>> We are making good headway with reworking the last parts of
> > the
> > > > > >>> Window
> > > > > >>>> API.
> > > > > >>>>>>> After that, the streaming API should be good to be pulled
> out
> > > of
> > > > > >>>> staging.
> > > > > >>>>>>>
> > > > > >>>>>>> Since we are reorganizing the projects as part of that, I
> > would
> > > > > shift
> > > > > >>>> a bit
> > > > > >>>>>>> more to bring things a bit more up to date.
> > > > > >>>>>>>
> > > > > >>>>>>> In this restructure, I would like to get rid of the
> > > > "flink-staging"
> > > > > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > > > > difference
> > > > > >>>>>>> whether a project is in "staging" or not, so it does not
> help
> > > > much
> > > > > to
> > > > > >>>> have
> > > > > >>>>>>> that directory structure.
> > > > > >>>>>>> On the other hand, projects have a tendency to linger in
> > > staging
> > > > > >>>> forever
> > > > > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > > > > >>>>>>>
> > > > > >>>>>>> The new structure could be
> > > > > >>>>>>>
> > > > > >>>>>>> flink-core
> > > > > >>>>>>> flink-java
> > > > > >>>>>>> flink-scala
> > > > > >>>>>>> flink-streaming-core
> > > > > >>>>>>> flink-streaming-scala
> > > > > >>>>>>>
> > > > > >>>>>>> flink-runtime
> > > > > >>>>>>> flink-runtime-web
> > > > > >>>>>>> flink-optimizer
> > > > > >>>>>>> flink-clients
> > > > > >>>>>>>
> > > > > >>>>>>> flink-shaded
> > > > > >>>>>>> -> flink-shaded-hadoop
> > > > > >>>>>>> -> flink-shaded-hadoop2
> > > > > >>>>>>> -> flink-shaded-include-yarn-tests
> > > > > >>>>>>> -> flink-shaded-curator
> > > > > >>>>>>>
> > > > > >>>>>>> flink-examples
> > > > > >>>>>>> -> (have all examples, Scala and Java, Batch and
> Streaming)
> > > > > >>>>>>>
> > > > > >>>>>>> flink-batch-connectors
> > > > > >>>>>>> -> flink-avro
> > > > > >>>>>>> -> flink-jdbc
> > > > > >>>>>>> -> flink-hadoop-compatibility
> > > > > >>>>>>> -> flink-hbase
> > > > > >>>>>>> -> flink-hcatalog
> > > > > >>>>>>>
> > > > > >>>>>>> flink-streaming-connectors
> > > > > >>>>>>> -> flink-connector-twitter
> > > > > >>>>>>> -> flink-streaming-examples
> > > > > >>>>>>> -> flink-connector-flume
> > > > > >>>>>>> -> flink-connector-kafka
> > > > > >>>>>>> -> flink-connector-elasticsearch
> > > > > >>>>>>> -> flink-connector-rabbitmq
> > > > > >>>>>>> -> flink-connector-filesystem
> > > > > >>>>>>>
> > > > > >>>>>>> flink-libraries
> > > > > >>>>>>> -> flink-gelly
> > > > > >>>>>>> -> flink-gelly-scala
> > > > > >>>>>>> -> flink-ml
> > > > > >>>>>>> -> flink-table
> > > > > >>>>>>> -> flink-language-binding
> > > > > >>>>>>> -> flink-python
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> flink-scala-shell
> > > > > >>>>>>>
> > > > > >>>>>>> flink-test-utils
> > > > > >>>>>>> flink-tests
> > > > > >>>>>>> flink-fs-tests
> > > > > >>>>>>>
> > > > > >>>>>>> flink-contrib
> > > > > >>>>>>> -> flink-storm-compatibility
> > > > > >>>>>>> -> flink-storm-compatibility-examples
> > > > > >>>>>>> -> flink-streaming-utils
> > > > > >>>>>>> -> flink-tweet-inputformat
> > > > > >>>>>>> -> flink-operator-stats
> > > > > >>>>>>> -> flink-tez
> > > > > >>>>>>>
> > > > > >>>>>>> flink-quickstart
> > > > > >>>>>>> -> flink-quickstart-java
> > > > > >>>>>>> -> flink-quickstart-scala
> > > > > >>>>>>> -> flink-tez-quickstart
> > > > > >>>>>>>
> > > > > >>>>>>> flink-yarn
> > > > > >>>>>>> flink-yarn-tests
> > > > > >>>>>>>
> > > > > >>>>>>> flink-dist
> > > > > >>>>>>>
> > > > > >>>>>>> flink-benchmark
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Let me know if that makes sense!
> > > > > >>>>>>>
> > > > > >>>>>>> Greetings,
> > > > > >>>>>>> Stephan
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>

Flavio Pompermaier

Re: Pulling Streaming out of staging and project restructure

Yes I know that right nowthat's to way to compile Flink without tests but
from my perspective (as a developer) that's a workaround. If I don't want
to compile the test classes I should be able to skip it (also because in
Flink compiling tests take quite a lot).
Of course it's not a big deal, it's just a matter of following Maven
principle of convention-over-configuration, so test sources should go in
test folders/jars.

On Mon, Jan 11, 2016 at 1:54 PM, Stephan Ewen <[hidden email]> wrote:

> Hi Flavio!
>
> You can always do "mvn -DskipTests clean package". That compiles tests, but
> does not execute them.
>
> Stephan
>
>
> On Mon, Jan 11, 2016 at 11:34 AM, Flavio Pompermaier <[hidden email]
> >
> wrote:
>
> > I hope that it's not too late to suggest to add to restructuring also
> > https://issues.apache.org/jira/browse/FLINK-1827 so that to be able to
> > compile Flink without compiling also tests (-Dmaven.test.skip=true) and
> > save a lot of time...
> > I should be fairly easy to fix that.
> >
> > Best,
> > Flavio
> >
> > On Wed, Jan 6, 2016 at 4:22 PM, Fabian Hueske <[hidden email]> wrote:
> >
> > > Stephan already opened a PR for restructuring the examples:
> > >
> > > --> https://github.com/apache/flink/pull/1482
> > >
> > > Otherwise +1!
> > >
> > > 2016-01-06 16:21 GMT+01:00 Robert Metzger <[hidden email]>:
> > >
> > > > The renaming hasn't been done completely. I would like to fix this
> > before
> > > > the 1.0.0 release.
> > > >
> > > > I think the following issues are still open:
> > > > - Merge flink-java-examples, flink-scala-examples and
> > > > flink-streaming-examples into "flink-examples" + build the streaming
> > > > examples for the binary build as well.
> > > > - move flink-table to flink-libraries
> > > > - move some modules from flink-staging to flink-batch-connectors
> > > >
> > > >
> > > > If nobody disagrees, I'll open a PR for these changes.
> > > > I created a JIRA for this discussion:
> > > > https://issues.apache.org/jira/browse/FLINK-3205
> > > >
> > > >
> > > > On Fri, Oct 2, 2015 at 7:58 PM, fhueske <[hidden email]> wrote:
> > > >
> > > > > +1
> > > > >
> > > > >
> > > > > From: Henry Saputra
> > > > > Sent: Friday, October 2, 2015 19:34
> > > > > To: [hidden email]
> > > > > Subject: Re: Pulling Streaming out of staging and project
> restructure
> > > > >
> > > > >
> > > > > +1
> > > > >
> > > > > On Friday, October 2, 2015, Matthias J. Sax <[hidden email]>
> > wrote:
> > > > >
> > > > > > It think, rename "flink-storm-compatibility-core" to just
> > > "flink-storm"
> > > > > > would be the cleanest solution.
> > > > > >
> > > > > > So in flink-contrib there would be two modules:
> > > > > > - flink-storm
> > > > > > - flink-storm-examples
> > > > > >
> > > > > > Please let me know if you have any objection about it.
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > > > > > > Sure. Will do that.
> > > > > > >
> > > > > > > -Matthias
> > > > > > >
> > > > > > > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> > > > > > >> @Matthias: How about getting rid of the
> > storm-compatibility-parent
> > > > and
> > > > > > >> making the core and examples projects directly projects in
> > > "contrib"
> > > > > > >>
> > > > > > >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <
> > > > [hidden email]
> > > > > > <javascript:;>> wrote:
> > > > > > >>
> > > > > > >>> +1 for the new project structure. Getting rid of our code
> dump
> > > is a
> > > > > > good
> > > > > > >>> thing.
> > > > > > >>>
> > > > > > >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <
> > > > [hidden email]
> > > > > > <javascript:;>>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> +1 Matthias, let's limit the overhead this has for the
> module
> > > > > > >>> maintainers.
> > > > > > >>>>
> > > > > > >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <
> > > > [hidden email]
> > > > > > <javascript:;>>
> > > > > > >>> wrote:
> > > > > > >>>>> I will commit something to flink-storm-compatibility
> tomorrow
> > > > that
> > > > > > >>>>> contains some internal package restructuring. I think,
> > renaming
> > > > the
> > > > > > >>>>> three modules in this commit would be a smart move as both
> > > > changes
> > > > > > >>>>> result in merge conflicts when rebasing open PRs. Thus we
> can
> > > > limit
> > > > > > >>> this
> > > > > > >>>>> pain to a single time. If no objections, I will commit
> those
> > > > > changes
> > > > > > >>>>> tomorrow.
> > > > > > >>>>>
> > > > > > >>>>> -Matthias
> > > > > > >>>>>
> > > > > > >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > > > > > >>>>>> +1
> > > > > > >>>>>>
> > > > > > >>>>>> I like the idea moving "staging" projects into appropriate
> > > > > modules.
> > > > > > >>>>>>
> > > > > > >>>>>> While we are at it, I would like to propose changing "
> > > > > > >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my
> > > > bucket
> > > > > > list
> > > > > > >>>>>> but would be nice if it is part of re-org.
> > > > > > >>>>>> Supporting Hadoop in the connector implicitly means
> > > > compatibility
> > > > > > with
> > > > > > >>>> Hadoop.
> > > > > > >>>>>> Also same thing with "flink-storm-compatibility" to
> > > > "flink-storm".
> > > > > > >>>>>>
> > > > > > >>>>>> - Henry
> > > > > > >>>>>>
> > > > > > >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <
> > > [hidden email]
> > > > > > <javascript:;>>
> > > > > > >>> wrote:
> > > > > > >>>>>>> Hi all!
> > > > > > >>>>>>>
> > > > > > >>>>>>> We are making good headway with reworking the last parts
> of
> > > the
> > > > > > >>> Window
> > > > > > >>>> API.
> > > > > > >>>>>>> After that, the streaming API should be good to be pulled
> > out
> > > > of
> > > > > > >>>> staging.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Since we are reorganizing the projects as part of that, I
> > > would
> > > > > > shift
> > > > > > >>>> a bit
> > > > > > >>>>>>> more to bring things a bit more up to date.
> > > > > > >>>>>>>
> > > > > > >>>>>>> In this restructure, I would like to get rid of the
> > > > > "flink-staging"
> > > > > > >>>>>>> project. Anyone who only uses the maven artifacts sees no
> > > > > > difference
> > > > > > >>>>>>> whether a project is in "staging" or not, so it does not
> > help
> > > > > much
> > > > > > to
> > > > > > >>>> have
> > > > > > >>>>>>> that directory structure.
> > > > > > >>>>>>> On the other hand, projects have a tendency to linger in
> > > > staging
> > > > > > >>>> forever
> > > > > > >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> > > > > > >>>>>>>
> > > > > > >>>>>>> The new structure could be
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-core
> > > > > > >>>>>>> flink-java
> > > > > > >>>>>>> flink-scala
> > > > > > >>>>>>> flink-streaming-core
> > > > > > >>>>>>> flink-streaming-scala
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-runtime
> > > > > > >>>>>>> flink-runtime-web
> > > > > > >>>>>>> flink-optimizer
> > > > > > >>>>>>> flink-clients
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-shaded
> > > > > > >>>>>>> -> flink-shaded-hadoop
> > > > > > >>>>>>> -> flink-shaded-hadoop2
> > > > > > >>>>>>> -> flink-shaded-include-yarn-tests
> > > > > > >>>>>>> -> flink-shaded-curator
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-examples
> > > > > > >>>>>>> -> (have all examples, Scala and Java, Batch and
> > Streaming)
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-batch-connectors
> > > > > > >>>>>>> -> flink-avro
> > > > > > >>>>>>> -> flink-jdbc
> > > > > > >>>>>>> -> flink-hadoop-compatibility
> > > > > > >>>>>>> -> flink-hbase
> > > > > > >>>>>>> -> flink-hcatalog
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-streaming-connectors
> > > > > > >>>>>>> -> flink-connector-twitter
> > > > > > >>>>>>> -> flink-streaming-examples
> > > > > > >>>>>>> -> flink-connector-flume
> > > > > > >>>>>>> -> flink-connector-kafka
> > > > > > >>>>>>> -> flink-connector-elasticsearch
> > > > > > >>>>>>> -> flink-connector-rabbitmq
> > > > > > >>>>>>> -> flink-connector-filesystem
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-libraries
> > > > > > >>>>>>> -> flink-gelly
> > > > > > >>>>>>> -> flink-gelly-scala
> > > > > > >>>>>>> -> flink-ml
> > > > > > >>>>>>> -> flink-table
> > > > > > >>>>>>> -> flink-language-binding
> > > > > > >>>>>>> -> flink-python
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-scala-shell
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-test-utils
> > > > > > >>>>>>> flink-tests
> > > > > > >>>>>>> flink-fs-tests
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-contrib
> > > > > > >>>>>>> -> flink-storm-compatibility
> > > > > > >>>>>>> -> flink-storm-compatibility-examples
> > > > > > >>>>>>> -> flink-streaming-utils
> > > > > > >>>>>>> -> flink-tweet-inputformat
> > > > > > >>>>>>> -> flink-operator-stats
> > > > > > >>>>>>> -> flink-tez
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-quickstart
> > > > > > >>>>>>> -> flink-quickstart-java
> > > > > > >>>>>>> -> flink-quickstart-scala
> > > > > > >>>>>>> -> flink-tez-quickstart
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-yarn
> > > > > > >>>>>>> flink-yarn-tests
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-dist
> > > > > > >>>>>>>
> > > > > > >>>>>>> flink-benchmark
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> Let me know if that makes sense!
> > > > > > >>>>>>>
> > > > > > >>>>>>> Greetings,
> > > > > > >>>>>>> Stephan
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>