Pulling Streaming out of staging and project restructure

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Pulling Streaming out of staging and project restructure

Stephan Ewen
Hi all!

We are making good headway with reworking the last parts of the Window API.
After that, the streaming API should be good to be pulled out of staging.

Since we are reorganizing the projects as part of that, I would shift a bit
more to bring things a bit more up to date.

In this restructure, I would like to get rid of the "flink-staging"
project. Anyone who only uses the maven artifacts sees no difference
whether a project is in "staging" or not, so it does not help much to have
that directory structure.
On the other hand, projects have a tendency to linger in staging forever
(like avro, spargel, hbase, jdbc, ...)

The new structure could be

flink-core
flink-java
flink-scala
flink-streaming-core
flink-streaming-scala

flink-runtime
flink-runtime-web
flink-optimizer
flink-clients

flink-shaded
  -> flink-shaded-hadoop
  -> flink-shaded-hadoop2
  -> flink-shaded-include-yarn-tests
  -> flink-shaded-curator

flink-examples
  -> (have all examples, Scala and Java, Batch and Streaming)

flink-batch-connectors
  -> flink-avro
  -> flink-jdbc
  -> flink-hadoop-compatibility
  -> flink-hbase
  -> flink-hcatalog

flink-streaming-connectors
  -> flink-connector-twitter
  -> flink-streaming-examples
  -> flink-connector-flume
  -> flink-connector-kafka
  -> flink-connector-elasticsearch
  -> flink-connector-rabbitmq
  -> flink-connector-filesystem

flink-libraries
  -> flink-gelly
  -> flink-gelly-scala
  -> flink-ml
  -> flink-table
  -> flink-language-binding
  -> flink-python


flink-scala-shell

flink-test-utils
flink-tests
flink-fs-tests

flink-contrib
  -> flink-storm-compatibility
  -> flink-storm-compatibility-examples
  -> flink-streaming-utils
  -> flink-tweet-inputformat
  -> flink-operator-stats
  -> flink-tez

flink-quickstart
  -> flink-quickstart-java
  -> flink-quickstart-scala
  -> flink-tez-quickstart

flink-yarn
flink-yarn-tests

flink-dist

flink-benchmark


Let me know if that makes sense!

Greetings,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Márton Balassi
Great to see streaming graduating. :)

I like the outline, both getting rid of staging, having the examples
together and generally flattening the structure are very reasonable to me.

You have listed flink-streaming-examples under flink-streaming-connectors
and left out some less prominent maven modules, but I assume the first is
accidental while the second is intentional to make the list a bit briefer.

Best,

Marton


On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]> wrote:

> Hi all!
>
> We are making good headway with reworking the last parts of the Window API.
> After that, the streaming API should be good to be pulled out of staging.
>
> Since we are reorganizing the projects as part of that, I would shift a bit
> more to bring things a bit more up to date.
>
> In this restructure, I would like to get rid of the "flink-staging"
> project. Anyone who only uses the maven artifacts sees no difference
> whether a project is in "staging" or not, so it does not help much to have
> that directory structure.
> On the other hand, projects have a tendency to linger in staging forever
> (like avro, spargel, hbase, jdbc, ...)
>
> The new structure could be
>
> flink-core
> flink-java
> flink-scala
> flink-streaming-core
> flink-streaming-scala
>
> flink-runtime
> flink-runtime-web
> flink-optimizer
> flink-clients
>
> flink-shaded
>   -> flink-shaded-hadoop
>   -> flink-shaded-hadoop2
>   -> flink-shaded-include-yarn-tests
>   -> flink-shaded-curator
>
> flink-examples
>   -> (have all examples, Scala and Java, Batch and Streaming)
>
> flink-batch-connectors
>   -> flink-avro
>   -> flink-jdbc
>   -> flink-hadoop-compatibility
>   -> flink-hbase
>   -> flink-hcatalog
>
> flink-streaming-connectors
>   -> flink-connector-twitter
>   -> flink-streaming-examples
>   -> flink-connector-flume
>   -> flink-connector-kafka
>   -> flink-connector-elasticsearch
>   -> flink-connector-rabbitmq
>   -> flink-connector-filesystem
>
> flink-libraries
>   -> flink-gelly
>   -> flink-gelly-scala
>   -> flink-ml
>   -> flink-table
>   -> flink-language-binding
>   -> flink-python
>
>
> flink-scala-shell
>
> flink-test-utils
> flink-tests
> flink-fs-tests
>
> flink-contrib
>   -> flink-storm-compatibility
>   -> flink-storm-compatibility-examples
>   -> flink-streaming-utils
>   -> flink-tweet-inputformat
>   -> flink-operator-stats
>   -> flink-tez
>
> flink-quickstart
>   -> flink-quickstart-java
>   -> flink-quickstart-scala
>   -> flink-tez-quickstart
>
> flink-yarn
> flink-yarn-tests
>
> flink-dist
>
> flink-benchmark
>
>
> Let me know if that makes sense!
>
> Greetings,
> Stephan
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Chesnay Schepler-3
I like it in general. But while we're at it, what is the purpose of the
flink-tests project, or rather which tests belong there instead of the
individual projects?

Where would new projects reside in, that previously would have been put
into flink-staging?

Lastly, I'd like to merge flink-language-binding into flink-python. I
can go more into detail but the gist of it is that the abstraction just
doesn't work.

On 01.10.2015 12:40, Márton Balassi wrote:

> Great to see streaming graduating. :)
>
> I like the outline, both getting rid of staging, having the examples
> together and generally flattening the structure are very reasonable to me.
>
> You have listed flink-streaming-examples under flink-streaming-connectors
> and left out some less prominent maven modules, but I assume the first is
> accidental while the second is intentional to make the list a bit briefer.
>
> Best,
>
> Marton
>
>
> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]> wrote:
>
>> Hi all!
>>
>> We are making good headway with reworking the last parts of the Window API.
>> After that, the streaming API should be good to be pulled out of staging.
>>
>> Since we are reorganizing the projects as part of that, I would shift a bit
>> more to bring things a bit more up to date.
>>
>> In this restructure, I would like to get rid of the "flink-staging"
>> project. Anyone who only uses the maven artifacts sees no difference
>> whether a project is in "staging" or not, so it does not help much to have
>> that directory structure.
>> On the other hand, projects have a tendency to linger in staging forever
>> (like avro, spargel, hbase, jdbc, ...)
>>
>> The new structure could be
>>
>> flink-core
>> flink-java
>> flink-scala
>> flink-streaming-core
>> flink-streaming-scala
>>
>> flink-runtime
>> flink-runtime-web
>> flink-optimizer
>> flink-clients
>>
>> flink-shaded
>>    -> flink-shaded-hadoop
>>    -> flink-shaded-hadoop2
>>    -> flink-shaded-include-yarn-tests
>>    -> flink-shaded-curator
>>
>> flink-examples
>>    -> (have all examples, Scala and Java, Batch and Streaming)
>>
>> flink-batch-connectors
>>    -> flink-avro
>>    -> flink-jdbc
>>    -> flink-hadoop-compatibility
>>    -> flink-hbase
>>    -> flink-hcatalog
>>
>> flink-streaming-connectors
>>    -> flink-connector-twitter
>>    -> flink-streaming-examples
>>    -> flink-connector-flume
>>    -> flink-connector-kafka
>>    -> flink-connector-elasticsearch
>>    -> flink-connector-rabbitmq
>>    -> flink-connector-filesystem
>>
>> flink-libraries
>>    -> flink-gelly
>>    -> flink-gelly-scala
>>    -> flink-ml
>>    -> flink-table
>>    -> flink-language-binding
>>    -> flink-python
>>
>>
>> flink-scala-shell
>>
>> flink-test-utils
>> flink-tests
>> flink-fs-tests
>>
>> flink-contrib
>>    -> flink-storm-compatibility
>>    -> flink-storm-compatibility-examples
>>    -> flink-streaming-utils
>>    -> flink-tweet-inputformat
>>    -> flink-operator-stats
>>    -> flink-tez
>>
>> flink-quickstart
>>    -> flink-quickstart-java
>>    -> flink-quickstart-scala
>>    -> flink-tez-quickstart
>>
>> flink-yarn
>> flink-yarn-tests
>>
>> flink-dist
>>
>> flink-benchmark
>>
>>
>> Let me know if that makes sense!
>>
>> Greetings,
>> Stephan
>>

Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Kostas Tzoumas-2
+1

I wanted to suggest that we rename modules to fully accept streaming as
first class, qualifying also "batch" as "batch" (e.g., flink-java -->
flink-dataset-java, flink-streaming --> flink-datastream, etc).

This would break maven dependencies (temporary hell :-) so it's not a
decision to take lightly. I'm not strongly advocating for it.


On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <[hidden email]>
wrote:

> I like it in general. But while we're at it, what is the purpose of the
> flink-tests project, or rather which tests belong there instead of the
> individual projects?
>
> Where would new projects reside in, that previously would have been put
> into flink-staging?
>
> Lastly, I'd like to merge flink-language-binding into flink-python. I can
> go more into detail but the gist of it is that the abstraction just doesn't
> work.
>
>
> On 01.10.2015 12:40, Márton Balassi wrote:
>
>> Great to see streaming graduating. :)
>>
>> I like the outline, both getting rid of staging, having the examples
>> together and generally flattening the structure are very reasonable to me.
>>
>> You have listed flink-streaming-examples under flink-streaming-connectors
>> and left out some less prominent maven modules, but I assume the first is
>> accidental while the second is intentional to make the list a bit briefer.
>>
>> Best,
>>
>> Marton
>>
>>
>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]> wrote:
>>
>> Hi all!
>>>
>>> We are making good headway with reworking the last parts of the Window
>>> API.
>>> After that, the streaming API should be good to be pulled out of staging.
>>>
>>> Since we are reorganizing the projects as part of that, I would shift a
>>> bit
>>> more to bring things a bit more up to date.
>>>
>>> In this restructure, I would like to get rid of the "flink-staging"
>>> project. Anyone who only uses the maven artifacts sees no difference
>>> whether a project is in "staging" or not, so it does not help much to
>>> have
>>> that directory structure.
>>> On the other hand, projects have a tendency to linger in staging forever
>>> (like avro, spargel, hbase, jdbc, ...)
>>>
>>> The new structure could be
>>>
>>> flink-core
>>> flink-java
>>> flink-scala
>>> flink-streaming-core
>>> flink-streaming-scala
>>>
>>> flink-runtime
>>> flink-runtime-web
>>> flink-optimizer
>>> flink-clients
>>>
>>> flink-shaded
>>>    -> flink-shaded-hadoop
>>>    -> flink-shaded-hadoop2
>>>    -> flink-shaded-include-yarn-tests
>>>    -> flink-shaded-curator
>>>
>>> flink-examples
>>>    -> (have all examples, Scala and Java, Batch and Streaming)
>>>
>>> flink-batch-connectors
>>>    -> flink-avro
>>>    -> flink-jdbc
>>>    -> flink-hadoop-compatibility
>>>    -> flink-hbase
>>>    -> flink-hcatalog
>>>
>>> flink-streaming-connectors
>>>    -> flink-connector-twitter
>>>    -> flink-streaming-examples
>>>    -> flink-connector-flume
>>>    -> flink-connector-kafka
>>>    -> flink-connector-elasticsearch
>>>    -> flink-connector-rabbitmq
>>>    -> flink-connector-filesystem
>>>
>>> flink-libraries
>>>    -> flink-gelly
>>>    -> flink-gelly-scala
>>>    -> flink-ml
>>>    -> flink-table
>>>    -> flink-language-binding
>>>    -> flink-python
>>>
>>>
>>> flink-scala-shell
>>>
>>> flink-test-utils
>>> flink-tests
>>> flink-fs-tests
>>>
>>> flink-contrib
>>>    -> flink-storm-compatibility
>>>    -> flink-storm-compatibility-examples
>>>    -> flink-streaming-utils
>>>    -> flink-tweet-inputformat
>>>    -> flink-operator-stats
>>>    -> flink-tez
>>>
>>> flink-quickstart
>>>    -> flink-quickstart-java
>>>    -> flink-quickstart-scala
>>>    -> flink-tez-quickstart
>>>
>>> flink-yarn
>>> flink-yarn-tests
>>>
>>> flink-dist
>>>
>>> flink-benchmark
>>>
>>>
>>> Let me know if that makes sense!
>>>
>>> Greetings,
>>> Stephan
>>>
>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Robert Metzger
Big +1 for graduating streaming out of staging. It is widely used, also in
production and we are spending a lot of effort into hardening it.
I also agree with the proposed new maven module structure.

We have to carefully test the reworked structure for the scripts which are
generating the hadoop1 and the scala 2.11 poms (they are transformed using
a bunch of bash scripts). I can do that once the PR is open.

@Chesnay: I would be fine with including the language binding into python
> Where would new projects reside in, that previously would have been put
into flink-staging?

flink-contrib


@Kostas: I understand the idea behind your suggested renaming, but thats
just a name. I don't think its going to influence how people are seeing
Flink: It doesn't feel like second class when adding "flink-streaming-core"
to the dependencies to me.
Also, the "flink-datastream-scala" module would depend on
"flink-dataset-scala", which is kind of weird.


I'm wondering whether we should remove the "flink-test-utils" module. I
don't think its really necessary, because we can put the test jars into the
flink-tests project and include them using the "test-jar" dependency.


On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <[hidden email]> wrote:

> +1
>
> I wanted to suggest that we rename modules to fully accept streaming as
> first class, qualifying also "batch" as "batch" (e.g., flink-java -->
> flink-dataset-java, flink-streaming --> flink-datastream, etc).
>
> This would break maven dependencies (temporary hell :-) so it's not a
> decision to take lightly. I'm not strongly advocating for it.
>
>
> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <[hidden email]>
> wrote:
>
> > I like it in general. But while we're at it, what is the purpose of the
> > flink-tests project, or rather which tests belong there instead of the
> > individual projects?
> >
> > Where would new projects reside in, that previously would have been put
> > into flink-staging?
> >
> > Lastly, I'd like to merge flink-language-binding into flink-python. I can
> > go more into detail but the gist of it is that the abstraction just
> doesn't
> > work.
> >
> >
> > On 01.10.2015 12:40, Márton Balassi wrote:
> >
> >> Great to see streaming graduating. :)
> >>
> >> I like the outline, both getting rid of staging, having the examples
> >> together and generally flattening the structure are very reasonable to
> me.
> >>
> >> You have listed flink-streaming-examples under
> flink-streaming-connectors
> >> and left out some less prominent maven modules, but I assume the first
> is
> >> accidental while the second is intentional to make the list a bit
> briefer.
> >>
> >> Best,
> >>
> >> Marton
> >>
> >>
> >> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]> wrote:
> >>
> >> Hi all!
> >>>
> >>> We are making good headway with reworking the last parts of the Window
> >>> API.
> >>> After that, the streaming API should be good to be pulled out of
> staging.
> >>>
> >>> Since we are reorganizing the projects as part of that, I would shift a
> >>> bit
> >>> more to bring things a bit more up to date.
> >>>
> >>> In this restructure, I would like to get rid of the "flink-staging"
> >>> project. Anyone who only uses the maven artifacts sees no difference
> >>> whether a project is in "staging" or not, so it does not help much to
> >>> have
> >>> that directory structure.
> >>> On the other hand, projects have a tendency to linger in staging
> forever
> >>> (like avro, spargel, hbase, jdbc, ...)
> >>>
> >>> The new structure could be
> >>>
> >>> flink-core
> >>> flink-java
> >>> flink-scala
> >>> flink-streaming-core
> >>> flink-streaming-scala
> >>>
> >>> flink-runtime
> >>> flink-runtime-web
> >>> flink-optimizer
> >>> flink-clients
> >>>
> >>> flink-shaded
> >>>    -> flink-shaded-hadoop
> >>>    -> flink-shaded-hadoop2
> >>>    -> flink-shaded-include-yarn-tests
> >>>    -> flink-shaded-curator
> >>>
> >>> flink-examples
> >>>    -> (have all examples, Scala and Java, Batch and Streaming)
> >>>
> >>> flink-batch-connectors
> >>>    -> flink-avro
> >>>    -> flink-jdbc
> >>>    -> flink-hadoop-compatibility
> >>>    -> flink-hbase
> >>>    -> flink-hcatalog
> >>>
> >>> flink-streaming-connectors
> >>>    -> flink-connector-twitter
> >>>    -> flink-streaming-examples
> >>>    -> flink-connector-flume
> >>>    -> flink-connector-kafka
> >>>    -> flink-connector-elasticsearch
> >>>    -> flink-connector-rabbitmq
> >>>    -> flink-connector-filesystem
> >>>
> >>> flink-libraries
> >>>    -> flink-gelly
> >>>    -> flink-gelly-scala
> >>>    -> flink-ml
> >>>    -> flink-table
> >>>    -> flink-language-binding
> >>>    -> flink-python
> >>>
> >>>
> >>> flink-scala-shell
> >>>
> >>> flink-test-utils
> >>> flink-tests
> >>> flink-fs-tests
> >>>
> >>> flink-contrib
> >>>    -> flink-storm-compatibility
> >>>    -> flink-storm-compatibility-examples
> >>>    -> flink-streaming-utils
> >>>    -> flink-tweet-inputformat
> >>>    -> flink-operator-stats
> >>>    -> flink-tez
> >>>
> >>> flink-quickstart
> >>>    -> flink-quickstart-java
> >>>    -> flink-quickstart-scala
> >>>    -> flink-tez-quickstart
> >>>
> >>> flink-yarn
> >>> flink-yarn-tests
> >>>
> >>> flink-dist
> >>>
> >>> flink-benchmark
> >>>
> >>>
> >>> Let me know if that makes sense!
> >>>
> >>> Greetings,
> >>> Stephan
> >>>
> >>>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Stephan Ewen
+1 for Robert's comments.

On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <[hidden email]> wrote:

> Big +1 for graduating streaming out of staging. It is widely used, also in
> production and we are spending a lot of effort into hardening it.
> I also agree with the proposed new maven module structure.
>
> We have to carefully test the reworked structure for the scripts which are
> generating the hadoop1 and the scala 2.11 poms (they are transformed using
> a bunch of bash scripts). I can do that once the PR is open.
>
> @Chesnay: I would be fine with including the language binding into python
> > Where would new projects reside in, that previously would have been put
> into flink-staging?
>
> flink-contrib
>
>
> @Kostas: I understand the idea behind your suggested renaming, but thats
> just a name. I don't think its going to influence how people are seeing
> Flink: It doesn't feel like second class when adding "flink-streaming-core"
> to the dependencies to me.
> Also, the "flink-datastream-scala" module would depend on
> "flink-dataset-scala", which is kind of weird.
>
>
> I'm wondering whether we should remove the "flink-test-utils" module. I
> don't think its really necessary, because we can put the test jars into the
> flink-tests project and include them using the "test-jar" dependency.
>
>
> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <[hidden email]>
> wrote:
>
> > +1
> >
> > I wanted to suggest that we rename modules to fully accept streaming as
> > first class, qualifying also "batch" as "batch" (e.g., flink-java -->
> > flink-dataset-java, flink-streaming --> flink-datastream, etc).
> >
> > This would break maven dependencies (temporary hell :-) so it's not a
> > decision to take lightly. I'm not strongly advocating for it.
> >
> >
> > On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <[hidden email]>
> > wrote:
> >
> > > I like it in general. But while we're at it, what is the purpose of the
> > > flink-tests project, or rather which tests belong there instead of the
> > > individual projects?
> > >
> > > Where would new projects reside in, that previously would have been put
> > > into flink-staging?
> > >
> > > Lastly, I'd like to merge flink-language-binding into flink-python. I
> can
> > > go more into detail but the gist of it is that the abstraction just
> > doesn't
> > > work.
> > >
> > >
> > > On 01.10.2015 12:40, Márton Balassi wrote:
> > >
> > >> Great to see streaming graduating. :)
> > >>
> > >> I like the outline, both getting rid of staging, having the examples
> > >> together and generally flattening the structure are very reasonable to
> > me.
> > >>
> > >> You have listed flink-streaming-examples under
> > flink-streaming-connectors
> > >> and left out some less prominent maven modules, but I assume the first
> > is
> > >> accidental while the second is intentional to make the list a bit
> > briefer.
> > >>
> > >> Best,
> > >>
> > >> Marton
> > >>
> > >>
> > >> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]>
> wrote:
> > >>
> > >> Hi all!
> > >>>
> > >>> We are making good headway with reworking the last parts of the
> Window
> > >>> API.
> > >>> After that, the streaming API should be good to be pulled out of
> > staging.
> > >>>
> > >>> Since we are reorganizing the projects as part of that, I would
> shift a
> > >>> bit
> > >>> more to bring things a bit more up to date.
> > >>>
> > >>> In this restructure, I would like to get rid of the "flink-staging"
> > >>> project. Anyone who only uses the maven artifacts sees no difference
> > >>> whether a project is in "staging" or not, so it does not help much to
> > >>> have
> > >>> that directory structure.
> > >>> On the other hand, projects have a tendency to linger in staging
> > forever
> > >>> (like avro, spargel, hbase, jdbc, ...)
> > >>>
> > >>> The new structure could be
> > >>>
> > >>> flink-core
> > >>> flink-java
> > >>> flink-scala
> > >>> flink-streaming-core
> > >>> flink-streaming-scala
> > >>>
> > >>> flink-runtime
> > >>> flink-runtime-web
> > >>> flink-optimizer
> > >>> flink-clients
> > >>>
> > >>> flink-shaded
> > >>>    -> flink-shaded-hadoop
> > >>>    -> flink-shaded-hadoop2
> > >>>    -> flink-shaded-include-yarn-tests
> > >>>    -> flink-shaded-curator
> > >>>
> > >>> flink-examples
> > >>>    -> (have all examples, Scala and Java, Batch and Streaming)
> > >>>
> > >>> flink-batch-connectors
> > >>>    -> flink-avro
> > >>>    -> flink-jdbc
> > >>>    -> flink-hadoop-compatibility
> > >>>    -> flink-hbase
> > >>>    -> flink-hcatalog
> > >>>
> > >>> flink-streaming-connectors
> > >>>    -> flink-connector-twitter
> > >>>    -> flink-streaming-examples
> > >>>    -> flink-connector-flume
> > >>>    -> flink-connector-kafka
> > >>>    -> flink-connector-elasticsearch
> > >>>    -> flink-connector-rabbitmq
> > >>>    -> flink-connector-filesystem
> > >>>
> > >>> flink-libraries
> > >>>    -> flink-gelly
> > >>>    -> flink-gelly-scala
> > >>>    -> flink-ml
> > >>>    -> flink-table
> > >>>    -> flink-language-binding
> > >>>    -> flink-python
> > >>>
> > >>>
> > >>> flink-scala-shell
> > >>>
> > >>> flink-test-utils
> > >>> flink-tests
> > >>> flink-fs-tests
> > >>>
> > >>> flink-contrib
> > >>>    -> flink-storm-compatibility
> > >>>    -> flink-storm-compatibility-examples
> > >>>    -> flink-streaming-utils
> > >>>    -> flink-tweet-inputformat
> > >>>    -> flink-operator-stats
> > >>>    -> flink-tez
> > >>>
> > >>> flink-quickstart
> > >>>    -> flink-quickstart-java
> > >>>    -> flink-quickstart-scala
> > >>>    -> flink-tez-quickstart
> > >>>
> > >>> flink-yarn
> > >>> flink-yarn-tests
> > >>>
> > >>> flink-dist
> > >>>
> > >>> flink-benchmark
> > >>>
> > >>>
> > >>> Let me know if that makes sense!
> > >>>
> > >>> Greetings,
> > >>> Stephan
> > >>>
> > >>>
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Chesnay Schepler-3
If we remove flink-staging because projects tend to get stuck there,
what mechanism prevents the same happening with flink-contrib?

On 01.10.2015 15:19, Stephan Ewen wrote:

> +1 for Robert's comments.
>
> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <[hidden email]> wrote:
>
>> Big +1 for graduating streaming out of staging. It is widely used, also in
>> production and we are spending a lot of effort into hardening it.
>> I also agree with the proposed new maven module structure.
>>
>> We have to carefully test the reworked structure for the scripts which are
>> generating the hadoop1 and the scala 2.11 poms (they are transformed using
>> a bunch of bash scripts). I can do that once the PR is open.
>>
>> @Chesnay: I would be fine with including the language binding into python
>>> Where would new projects reside in, that previously would have been put
>> into flink-staging?
>>
>> flink-contrib
>>
>>
>> @Kostas: I understand the idea behind your suggested renaming, but thats
>> just a name. I don't think its going to influence how people are seeing
>> Flink: It doesn't feel like second class when adding "flink-streaming-core"
>> to the dependencies to me.
>> Also, the "flink-datastream-scala" module would depend on
>> "flink-dataset-scala", which is kind of weird.
>>
>>
>> I'm wondering whether we should remove the "flink-test-utils" module. I
>> don't think its really necessary, because we can put the test jars into the
>> flink-tests project and include them using the "test-jar" dependency.
>>
>>
>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <[hidden email]>
>> wrote:
>>
>>> +1
>>>
>>> I wanted to suggest that we rename modules to fully accept streaming as
>>> first class, qualifying also "batch" as "batch" (e.g., flink-java -->
>>> flink-dataset-java, flink-streaming --> flink-datastream, etc).
>>>
>>> This would break maven dependencies (temporary hell :-) so it's not a
>>> decision to take lightly. I'm not strongly advocating for it.
>>>
>>>
>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <[hidden email]>
>>> wrote:
>>>
>>>> I like it in general. But while we're at it, what is the purpose of the
>>>> flink-tests project, or rather which tests belong there instead of the
>>>> individual projects?
>>>>
>>>> Where would new projects reside in, that previously would have been put
>>>> into flink-staging?
>>>>
>>>> Lastly, I'd like to merge flink-language-binding into flink-python. I
>> can
>>>> go more into detail but the gist of it is that the abstraction just
>>> doesn't
>>>> work.
>>>>
>>>>
>>>> On 01.10.2015 12:40, Márton Balassi wrote:
>>>>
>>>>> Great to see streaming graduating. :)
>>>>>
>>>>> I like the outline, both getting rid of staging, having the examples
>>>>> together and generally flattening the structure are very reasonable to
>>> me.
>>>>> You have listed flink-streaming-examples under
>>> flink-streaming-connectors
>>>>> and left out some less prominent maven modules, but I assume the first
>>> is
>>>>> accidental while the second is intentional to make the list a bit
>>> briefer.
>>>>> Best,
>>>>>
>>>>> Marton
>>>>>
>>>>>
>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]>
>> wrote:
>>>>> Hi all!
>>>>>> We are making good headway with reworking the last parts of the
>> Window
>>>>>> API.
>>>>>> After that, the streaming API should be good to be pulled out of
>>> staging.
>>>>>> Since we are reorganizing the projects as part of that, I would
>> shift a
>>>>>> bit
>>>>>> more to bring things a bit more up to date.
>>>>>>
>>>>>> In this restructure, I would like to get rid of the "flink-staging"
>>>>>> project. Anyone who only uses the maven artifacts sees no difference
>>>>>> whether a project is in "staging" or not, so it does not help much to
>>>>>> have
>>>>>> that directory structure.
>>>>>> On the other hand, projects have a tendency to linger in staging
>>> forever
>>>>>> (like avro, spargel, hbase, jdbc, ...)
>>>>>>
>>>>>> The new structure could be
>>>>>>
>>>>>> flink-core
>>>>>> flink-java
>>>>>> flink-scala
>>>>>> flink-streaming-core
>>>>>> flink-streaming-scala
>>>>>>
>>>>>> flink-runtime
>>>>>> flink-runtime-web
>>>>>> flink-optimizer
>>>>>> flink-clients
>>>>>>
>>>>>> flink-shaded
>>>>>>     -> flink-shaded-hadoop
>>>>>>     -> flink-shaded-hadoop2
>>>>>>     -> flink-shaded-include-yarn-tests
>>>>>>     -> flink-shaded-curator
>>>>>>
>>>>>> flink-examples
>>>>>>     -> (have all examples, Scala and Java, Batch and Streaming)
>>>>>>
>>>>>> flink-batch-connectors
>>>>>>     -> flink-avro
>>>>>>     -> flink-jdbc
>>>>>>     -> flink-hadoop-compatibility
>>>>>>     -> flink-hbase
>>>>>>     -> flink-hcatalog
>>>>>>
>>>>>> flink-streaming-connectors
>>>>>>     -> flink-connector-twitter
>>>>>>     -> flink-streaming-examples
>>>>>>     -> flink-connector-flume
>>>>>>     -> flink-connector-kafka
>>>>>>     -> flink-connector-elasticsearch
>>>>>>     -> flink-connector-rabbitmq
>>>>>>     -> flink-connector-filesystem
>>>>>>
>>>>>> flink-libraries
>>>>>>     -> flink-gelly
>>>>>>     -> flink-gelly-scala
>>>>>>     -> flink-ml
>>>>>>     -> flink-table
>>>>>>     -> flink-language-binding
>>>>>>     -> flink-python
>>>>>>
>>>>>>
>>>>>> flink-scala-shell
>>>>>>
>>>>>> flink-test-utils
>>>>>> flink-tests
>>>>>> flink-fs-tests
>>>>>>
>>>>>> flink-contrib
>>>>>>     -> flink-storm-compatibility
>>>>>>     -> flink-storm-compatibility-examples
>>>>>>     -> flink-streaming-utils
>>>>>>     -> flink-tweet-inputformat
>>>>>>     -> flink-operator-stats
>>>>>>     -> flink-tez
>>>>>>
>>>>>> flink-quickstart
>>>>>>     -> flink-quickstart-java
>>>>>>     -> flink-quickstart-scala
>>>>>>     -> flink-tez-quickstart
>>>>>>
>>>>>> flink-yarn
>>>>>> flink-yarn-tests
>>>>>>
>>>>>> flink-dist
>>>>>>
>>>>>> flink-benchmark
>>>>>>
>>>>>>
>>>>>> Let me know if that makes sense!
>>>>>>
>>>>>> Greetings,
>>>>>> Stephan
>>>>>>
>>>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Kostas Tzoumas-2
+1 to Robert and practicality :-)

As I said before, I do not feel strongly about this, I was torn myself.

On Thu, Oct 1, 2015 at 3:28 PM, Chesnay Schepler <[hidden email]> wrote:

> If we remove flink-staging because projects tend to get stuck there, what
> mechanism prevents the same happening with flink-contrib?
>
>
> On 01.10.2015 15:19, Stephan Ewen wrote:
>
>> +1 for Robert's comments.
>>
>> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <[hidden email]>
>> wrote:
>>
>> Big +1 for graduating streaming out of staging. It is widely used, also in
>>> production and we are spending a lot of effort into hardening it.
>>> I also agree with the proposed new maven module structure.
>>>
>>> We have to carefully test the reworked structure for the scripts which
>>> are
>>> generating the hadoop1 and the scala 2.11 poms (they are transformed
>>> using
>>> a bunch of bash scripts). I can do that once the PR is open.
>>>
>>> @Chesnay: I would be fine with including the language binding into python
>>>
>>>> Where would new projects reside in, that previously would have been put
>>>>
>>> into flink-staging?
>>>
>>> flink-contrib
>>>
>>>
>>> @Kostas: I understand the idea behind your suggested renaming, but thats
>>> just a name. I don't think its going to influence how people are seeing
>>> Flink: It doesn't feel like second class when adding
>>> "flink-streaming-core"
>>> to the dependencies to me.
>>> Also, the "flink-datastream-scala" module would depend on
>>> "flink-dataset-scala", which is kind of weird.
>>>
>>>
>>> I'm wondering whether we should remove the "flink-test-utils" module. I
>>> don't think its really necessary, because we can put the test jars into
>>> the
>>> flink-tests project and include them using the "test-jar" dependency.
>>>
>>>
>>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <[hidden email]>
>>> wrote:
>>>
>>> +1
>>>>
>>>> I wanted to suggest that we rename modules to fully accept streaming as
>>>> first class, qualifying also "batch" as "batch" (e.g., flink-java -->
>>>> flink-dataset-java, flink-streaming --> flink-datastream, etc).
>>>>
>>>> This would break maven dependencies (temporary hell :-) so it's not a
>>>> decision to take lightly. I'm not strongly advocating for it.
>>>>
>>>>
>>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <[hidden email]>
>>>> wrote:
>>>>
>>>> I like it in general. But while we're at it, what is the purpose of the
>>>>> flink-tests project, or rather which tests belong there instead of the
>>>>> individual projects?
>>>>>
>>>>> Where would new projects reside in, that previously would have been put
>>>>> into flink-staging?
>>>>>
>>>>> Lastly, I'd like to merge flink-language-binding into flink-python. I
>>>>>
>>>> can
>>>
>>>> go more into detail but the gist of it is that the abstraction just
>>>>>
>>>> doesn't
>>>>
>>>>> work.
>>>>>
>>>>>
>>>>> On 01.10.2015 12:40, Márton Balassi wrote:
>>>>>
>>>>> Great to see streaming graduating. :)
>>>>>>
>>>>>> I like the outline, both getting rid of staging, having the examples
>>>>>> together and generally flattening the structure are very reasonable to
>>>>>>
>>>>> me.
>>>>
>>>>> You have listed flink-streaming-examples under
>>>>>>
>>>>> flink-streaming-connectors
>>>>
>>>>> and left out some less prominent maven modules, but I assume the first
>>>>>>
>>>>> is
>>>>
>>>>> accidental while the second is intentional to make the list a bit
>>>>>>
>>>>> briefer.
>>>>
>>>>> Best,
>>>>>>
>>>>>> Marton
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]>
>>>>>>
>>>>> wrote:
>>>
>>>> Hi all!
>>>>>>
>>>>>>> We are making good headway with reworking the last parts of the
>>>>>>>
>>>>>> Window
>>>
>>>> API.
>>>>>>> After that, the streaming API should be good to be pulled out of
>>>>>>>
>>>>>> staging.
>>>>
>>>>> Since we are reorganizing the projects as part of that, I would
>>>>>>>
>>>>>> shift a
>>>
>>>> bit
>>>>>>> more to bring things a bit more up to date.
>>>>>>>
>>>>>>> In this restructure, I would like to get rid of the "flink-staging"
>>>>>>> project. Anyone who only uses the maven artifacts sees no difference
>>>>>>> whether a project is in "staging" or not, so it does not help much to
>>>>>>> have
>>>>>>> that directory structure.
>>>>>>> On the other hand, projects have a tendency to linger in staging
>>>>>>>
>>>>>> forever
>>>>
>>>>> (like avro, spargel, hbase, jdbc, ...)
>>>>>>>
>>>>>>> The new structure could be
>>>>>>>
>>>>>>> flink-core
>>>>>>> flink-java
>>>>>>> flink-scala
>>>>>>> flink-streaming-core
>>>>>>> flink-streaming-scala
>>>>>>>
>>>>>>> flink-runtime
>>>>>>> flink-runtime-web
>>>>>>> flink-optimizer
>>>>>>> flink-clients
>>>>>>>
>>>>>>> flink-shaded
>>>>>>>     -> flink-shaded-hadoop
>>>>>>>     -> flink-shaded-hadoop2
>>>>>>>     -> flink-shaded-include-yarn-tests
>>>>>>>     -> flink-shaded-curator
>>>>>>>
>>>>>>> flink-examples
>>>>>>>     -> (have all examples, Scala and Java, Batch and Streaming)
>>>>>>>
>>>>>>> flink-batch-connectors
>>>>>>>     -> flink-avro
>>>>>>>     -> flink-jdbc
>>>>>>>     -> flink-hadoop-compatibility
>>>>>>>     -> flink-hbase
>>>>>>>     -> flink-hcatalog
>>>>>>>
>>>>>>> flink-streaming-connectors
>>>>>>>     -> flink-connector-twitter
>>>>>>>     -> flink-streaming-examples
>>>>>>>     -> flink-connector-flume
>>>>>>>     -> flink-connector-kafka
>>>>>>>     -> flink-connector-elasticsearch
>>>>>>>     -> flink-connector-rabbitmq
>>>>>>>     -> flink-connector-filesystem
>>>>>>>
>>>>>>> flink-libraries
>>>>>>>     -> flink-gelly
>>>>>>>     -> flink-gelly-scala
>>>>>>>     -> flink-ml
>>>>>>>     -> flink-table
>>>>>>>     -> flink-language-binding
>>>>>>>     -> flink-python
>>>>>>>
>>>>>>>
>>>>>>> flink-scala-shell
>>>>>>>
>>>>>>> flink-test-utils
>>>>>>> flink-tests
>>>>>>> flink-fs-tests
>>>>>>>
>>>>>>> flink-contrib
>>>>>>>     -> flink-storm-compatibility
>>>>>>>     -> flink-storm-compatibility-examples
>>>>>>>     -> flink-streaming-utils
>>>>>>>     -> flink-tweet-inputformat
>>>>>>>     -> flink-operator-stats
>>>>>>>     -> flink-tez
>>>>>>>
>>>>>>> flink-quickstart
>>>>>>>     -> flink-quickstart-java
>>>>>>>     -> flink-quickstart-scala
>>>>>>>     -> flink-tez-quickstart
>>>>>>>
>>>>>>> flink-yarn
>>>>>>> flink-yarn-tests
>>>>>>>
>>>>>>> flink-dist
>>>>>>>
>>>>>>> flink-benchmark
>>>>>>>
>>>>>>>
>>>>>>> Let me know if that makes sense!
>>>>>>>
>>>>>>> Greetings,
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Robert Metzger
@Chesnay: Nothing prevents projects from getting stuck there. But at least
we don't have two staging repositories (dist, staging). Also I would not
say that there has been no graduation out of staging. Yarn was also there
once, streaming and gelly are currently leaving it. So I would actually say
it worked.

One more thing: Can we actually put the web submission client into a
separate module until we have something new and better?

It does really not fit into "flink-clients". It makes Flink users pull the
jetty dependencies even if they never use them for example when running
Flink from an IDE.


On Thu, Oct 1, 2015 at 4:45 PM, Kostas Tzoumas <[hidden email]> wrote:

> +1 to Robert and practicality :-)
>
> As I said before, I do not feel strongly about this, I was torn myself.
>
> On Thu, Oct 1, 2015 at 3:28 PM, Chesnay Schepler <[hidden email]>
> wrote:
>
> > If we remove flink-staging because projects tend to get stuck there, what
> > mechanism prevents the same happening with flink-contrib?
> >
> >
> > On 01.10.2015 15:19, Stephan Ewen wrote:
> >
> >> +1 for Robert's comments.
> >>
> >> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <[hidden email]>
> >> wrote:
> >>
> >> Big +1 for graduating streaming out of staging. It is widely used, also
> in
> >>> production and we are spending a lot of effort into hardening it.
> >>> I also agree with the proposed new maven module structure.
> >>>
> >>> We have to carefully test the reworked structure for the scripts which
> >>> are
> >>> generating the hadoop1 and the scala 2.11 poms (they are transformed
> >>> using
> >>> a bunch of bash scripts). I can do that once the PR is open.
> >>>
> >>> @Chesnay: I would be fine with including the language binding into
> python
> >>>
> >>>> Where would new projects reside in, that previously would have been
> put
> >>>>
> >>> into flink-staging?
> >>>
> >>> flink-contrib
> >>>
> >>>
> >>> @Kostas: I understand the idea behind your suggested renaming, but
> thats
> >>> just a name. I don't think its going to influence how people are seeing
> >>> Flink: It doesn't feel like second class when adding
> >>> "flink-streaming-core"
> >>> to the dependencies to me.
> >>> Also, the "flink-datastream-scala" module would depend on
> >>> "flink-dataset-scala", which is kind of weird.
> >>>
> >>>
> >>> I'm wondering whether we should remove the "flink-test-utils" module. I
> >>> don't think its really necessary, because we can put the test jars into
> >>> the
> >>> flink-tests project and include them using the "test-jar" dependency.
> >>>
> >>>
> >>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <[hidden email]>
> >>> wrote:
> >>>
> >>> +1
> >>>>
> >>>> I wanted to suggest that we rename modules to fully accept streaming
> as
> >>>> first class, qualifying also "batch" as "batch" (e.g., flink-java -->
> >>>> flink-dataset-java, flink-streaming --> flink-datastream, etc).
> >>>>
> >>>> This would break maven dependencies (temporary hell :-) so it's not a
> >>>> decision to take lightly. I'm not strongly advocating for it.
> >>>>
> >>>>
> >>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <[hidden email]
> >
> >>>> wrote:
> >>>>
> >>>> I like it in general. But while we're at it, what is the purpose of
> the
> >>>>> flink-tests project, or rather which tests belong there instead of
> the
> >>>>> individual projects?
> >>>>>
> >>>>> Where would new projects reside in, that previously would have been
> put
> >>>>> into flink-staging?
> >>>>>
> >>>>> Lastly, I'd like to merge flink-language-binding into flink-python. I
> >>>>>
> >>>> can
> >>>
> >>>> go more into detail but the gist of it is that the abstraction just
> >>>>>
> >>>> doesn't
> >>>>
> >>>>> work.
> >>>>>
> >>>>>
> >>>>> On 01.10.2015 12:40, Márton Balassi wrote:
> >>>>>
> >>>>> Great to see streaming graduating. :)
> >>>>>>
> >>>>>> I like the outline, both getting rid of staging, having the examples
> >>>>>> together and generally flattening the structure are very reasonable
> to
> >>>>>>
> >>>>> me.
> >>>>
> >>>>> You have listed flink-streaming-examples under
> >>>>>>
> >>>>> flink-streaming-connectors
> >>>>
> >>>>> and left out some less prominent maven modules, but I assume the
> first
> >>>>>>
> >>>>> is
> >>>>
> >>>>> accidental while the second is intentional to make the list a bit
> >>>>>>
> >>>>> briefer.
> >>>>
> >>>>> Best,
> >>>>>>
> >>>>>> Marton
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <[hidden email]>
> >>>>>>
> >>>>> wrote:
> >>>
> >>>> Hi all!
> >>>>>>
> >>>>>>> We are making good headway with reworking the last parts of the
> >>>>>>>
> >>>>>> Window
> >>>
> >>>> API.
> >>>>>>> After that, the streaming API should be good to be pulled out of
> >>>>>>>
> >>>>>> staging.
> >>>>
> >>>>> Since we are reorganizing the projects as part of that, I would
> >>>>>>>
> >>>>>> shift a
> >>>
> >>>> bit
> >>>>>>> more to bring things a bit more up to date.
> >>>>>>>
> >>>>>>> In this restructure, I would like to get rid of the "flink-staging"
> >>>>>>> project. Anyone who only uses the maven artifacts sees no
> difference
> >>>>>>> whether a project is in "staging" or not, so it does not help much
> to
> >>>>>>> have
> >>>>>>> that directory structure.
> >>>>>>> On the other hand, projects have a tendency to linger in staging
> >>>>>>>
> >>>>>> forever
> >>>>
> >>>>> (like avro, spargel, hbase, jdbc, ...)
> >>>>>>>
> >>>>>>> The new structure could be
> >>>>>>>
> >>>>>>> flink-core
> >>>>>>> flink-java
> >>>>>>> flink-scala
> >>>>>>> flink-streaming-core
> >>>>>>> flink-streaming-scala
> >>>>>>>
> >>>>>>> flink-runtime
> >>>>>>> flink-runtime-web
> >>>>>>> flink-optimizer
> >>>>>>> flink-clients
> >>>>>>>
> >>>>>>> flink-shaded
> >>>>>>>     -> flink-shaded-hadoop
> >>>>>>>     -> flink-shaded-hadoop2
> >>>>>>>     -> flink-shaded-include-yarn-tests
> >>>>>>>     -> flink-shaded-curator
> >>>>>>>
> >>>>>>> flink-examples
> >>>>>>>     -> (have all examples, Scala and Java, Batch and Streaming)
> >>>>>>>
> >>>>>>> flink-batch-connectors
> >>>>>>>     -> flink-avro
> >>>>>>>     -> flink-jdbc
> >>>>>>>     -> flink-hadoop-compatibility
> >>>>>>>     -> flink-hbase
> >>>>>>>     -> flink-hcatalog
> >>>>>>>
> >>>>>>> flink-streaming-connectors
> >>>>>>>     -> flink-connector-twitter
> >>>>>>>     -> flink-streaming-examples
> >>>>>>>     -> flink-connector-flume
> >>>>>>>     -> flink-connector-kafka
> >>>>>>>     -> flink-connector-elasticsearch
> >>>>>>>     -> flink-connector-rabbitmq
> >>>>>>>     -> flink-connector-filesystem
> >>>>>>>
> >>>>>>> flink-libraries
> >>>>>>>     -> flink-gelly
> >>>>>>>     -> flink-gelly-scala
> >>>>>>>     -> flink-ml
> >>>>>>>     -> flink-table
> >>>>>>>     -> flink-language-binding
> >>>>>>>     -> flink-python
> >>>>>>>
> >>>>>>>
> >>>>>>> flink-scala-shell
> >>>>>>>
> >>>>>>> flink-test-utils
> >>>>>>> flink-tests
> >>>>>>> flink-fs-tests
> >>>>>>>
> >>>>>>> flink-contrib
> >>>>>>>     -> flink-storm-compatibility
> >>>>>>>     -> flink-storm-compatibility-examples
> >>>>>>>     -> flink-streaming-utils
> >>>>>>>     -> flink-tweet-inputformat
> >>>>>>>     -> flink-operator-stats
> >>>>>>>     -> flink-tez
> >>>>>>>
> >>>>>>> flink-quickstart
> >>>>>>>     -> flink-quickstart-java
> >>>>>>>     -> flink-quickstart-scala
> >>>>>>>     -> flink-tez-quickstart
> >>>>>>>
> >>>>>>> flink-yarn
> >>>>>>> flink-yarn-tests
> >>>>>>>
> >>>>>>> flink-dist
> >>>>>>>
> >>>>>>> flink-benchmark
> >>>>>>>
> >>>>>>>
> >>>>>>> Let me know if that makes sense!
> >>>>>>>
> >>>>>>> Greetings,
> >>>>>>> Stephan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Ufuk Celebi-2

> On 01 Oct 2015, at 16:48, Robert Metzger <[hidden email]> wrote:
>
> @Chesnay: Nothing prevents projects from getting stuck there. But at least
> we don't have two staging repositories (dist, staging). Also I would not
> say that there has been no graduation out of staging. Yarn was also there
> once, streaming and gelly are currently leaving it. So I would actually say
> it worked.
>
> One more thing: Can we actually put the web submission client into a
> separate module until we have something new and better?
>
> It does really not fit into "flink-clients". It makes Flink users pull the
> jetty dependencies even if they never use them for example when running
> Flink from an IDE.

I think most people use it as part of the binary distribution and nothing else anyways. So +1.
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Aljoscha Krettek-2
+1 For pulling out and the restructure. Enough good arguments have been
brought forward and I agree with all of them.

On Thu, 1 Oct 2015 at 17:47 Ufuk Celebi <[hidden email]> wrote:

>
> > On 01 Oct 2015, at 16:48, Robert Metzger <[hidden email]> wrote:
> >
> > @Chesnay: Nothing prevents projects from getting stuck there. But at
> least
> > we don't have two staging repositories (dist, staging). Also I would not
> > say that there has been no graduation out of staging. Yarn was also there
> > once, streaming and gelly are currently leaving it. So I would actually
> say
> > it worked.
> >
> > One more thing: Can we actually put the web submission client into a
> > separate module until we have something new and better?
> >
> > It does really not fit into "flink-clients". It makes Flink users pull
> the
> > jetty dependencies even if they never use them for example when running
> > Flink from an IDE.
>
> I think most people use it as part of the binary distribution and nothing
> else anyways. So +1.
mxm
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

mxm
+1 for the new Maven project structure
+1 for removing the flink-testing-utils module
+1 for moving flink-language-binding to flink-python

On Thu, Oct 1, 2015 at 6:27 PM, Aljoscha Krettek <[hidden email]> wrote:

> +1 For pulling out and the restructure. Enough good arguments have been
> brought forward and I agree with all of them.
>
> On Thu, 1 Oct 2015 at 17:47 Ufuk Celebi <[hidden email]> wrote:
>
>>
>> > On 01 Oct 2015, at 16:48, Robert Metzger <[hidden email]> wrote:
>> >
>> > @Chesnay: Nothing prevents projects from getting stuck there. But at
>> least
>> > we don't have two staging repositories (dist, staging). Also I would not
>> > say that there has been no graduation out of staging. Yarn was also there
>> > once, streaming and gelly are currently leaving it. So I would actually
>> say
>> > it worked.
>> >
>> > One more thing: Can we actually put the web submission client into a
>> > separate module until we have something new and better?
>> >
>> > It does really not fit into "flink-clients". It makes Flink users pull
>> the
>> > jetty dependencies even if they never use them for example when running
>> > Flink from an IDE.
>>
>> I think most people use it as part of the binary distribution and nothing
>> else anyways. So +1.
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Henry Saputra
In reply to this post by Stephan Ewen
+1

I like the idea moving "staging" projects into appropriate modules.

While we are at it, I would like to propose changing "
flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
but would be nice if it is part of re-org.
Supporting Hadoop in the connector implicitly means compatibility with Hadoop.
Also same thing with "flink-storm-compatibility" to "flink-storm".

- Henry

On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]> wrote:

> Hi all!
>
> We are making good headway with reworking the last parts of the Window API.
> After that, the streaming API should be good to be pulled out of staging.
>
> Since we are reorganizing the projects as part of that, I would shift a bit
> more to bring things a bit more up to date.
>
> In this restructure, I would like to get rid of the "flink-staging"
> project. Anyone who only uses the maven artifacts sees no difference
> whether a project is in "staging" or not, so it does not help much to have
> that directory structure.
> On the other hand, projects have a tendency to linger in staging forever
> (like avro, spargel, hbase, jdbc, ...)
>
> The new structure could be
>
> flink-core
> flink-java
> flink-scala
> flink-streaming-core
> flink-streaming-scala
>
> flink-runtime
> flink-runtime-web
> flink-optimizer
> flink-clients
>
> flink-shaded
>   -> flink-shaded-hadoop
>   -> flink-shaded-hadoop2
>   -> flink-shaded-include-yarn-tests
>   -> flink-shaded-curator
>
> flink-examples
>   -> (have all examples, Scala and Java, Batch and Streaming)
>
> flink-batch-connectors
>   -> flink-avro
>   -> flink-jdbc
>   -> flink-hadoop-compatibility
>   -> flink-hbase
>   -> flink-hcatalog
>
> flink-streaming-connectors
>   -> flink-connector-twitter
>   -> flink-streaming-examples
>   -> flink-connector-flume
>   -> flink-connector-kafka
>   -> flink-connector-elasticsearch
>   -> flink-connector-rabbitmq
>   -> flink-connector-filesystem
>
> flink-libraries
>   -> flink-gelly
>   -> flink-gelly-scala
>   -> flink-ml
>   -> flink-table
>   -> flink-language-binding
>   -> flink-python
>
>
> flink-scala-shell
>
> flink-test-utils
> flink-tests
> flink-fs-tests
>
> flink-contrib
>   -> flink-storm-compatibility
>   -> flink-storm-compatibility-examples
>   -> flink-streaming-utils
>   -> flink-tweet-inputformat
>   -> flink-operator-stats
>   -> flink-tez
>
> flink-quickstart
>   -> flink-quickstart-java
>   -> flink-quickstart-scala
>   -> flink-tez-quickstart
>
> flink-yarn
> flink-yarn-tests
>
> flink-dist
>
> flink-benchmark
>
>
> Let me know if that makes sense!
>
> Greetings,
> Stephan
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Matthias J. Sax-2
I will commit something to flink-storm-compatibility tomorrow that
contains some internal package restructuring. I think, renaming the
three modules in this commit would be a smart move as both changes
result in merge conflicts when rebasing open PRs. Thus we can limit this
pain to a single time. If no objections, I will commit those changes
tomorrow.

-Matthias

On 10/01/2015 09:52 PM, Henry Saputra wrote:

> +1
>
> I like the idea moving "staging" projects into appropriate modules.
>
> While we are at it, I would like to propose changing "
> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
> but would be nice if it is part of re-org.
> Supporting Hadoop in the connector implicitly means compatibility with Hadoop.
> Also same thing with "flink-storm-compatibility" to "flink-storm".
>
> - Henry
>
> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]> wrote:
>> Hi all!
>>
>> We are making good headway with reworking the last parts of the Window API.
>> After that, the streaming API should be good to be pulled out of staging.
>>
>> Since we are reorganizing the projects as part of that, I would shift a bit
>> more to bring things a bit more up to date.
>>
>> In this restructure, I would like to get rid of the "flink-staging"
>> project. Anyone who only uses the maven artifacts sees no difference
>> whether a project is in "staging" or not, so it does not help much to have
>> that directory structure.
>> On the other hand, projects have a tendency to linger in staging forever
>> (like avro, spargel, hbase, jdbc, ...)
>>
>> The new structure could be
>>
>> flink-core
>> flink-java
>> flink-scala
>> flink-streaming-core
>> flink-streaming-scala
>>
>> flink-runtime
>> flink-runtime-web
>> flink-optimizer
>> flink-clients
>>
>> flink-shaded
>>   -> flink-shaded-hadoop
>>   -> flink-shaded-hadoop2
>>   -> flink-shaded-include-yarn-tests
>>   -> flink-shaded-curator
>>
>> flink-examples
>>   -> (have all examples, Scala and Java, Batch and Streaming)
>>
>> flink-batch-connectors
>>   -> flink-avro
>>   -> flink-jdbc
>>   -> flink-hadoop-compatibility
>>   -> flink-hbase
>>   -> flink-hcatalog
>>
>> flink-streaming-connectors
>>   -> flink-connector-twitter
>>   -> flink-streaming-examples
>>   -> flink-connector-flume
>>   -> flink-connector-kafka
>>   -> flink-connector-elasticsearch
>>   -> flink-connector-rabbitmq
>>   -> flink-connector-filesystem
>>
>> flink-libraries
>>   -> flink-gelly
>>   -> flink-gelly-scala
>>   -> flink-ml
>>   -> flink-table
>>   -> flink-language-binding
>>   -> flink-python
>>
>>
>> flink-scala-shell
>>
>> flink-test-utils
>> flink-tests
>> flink-fs-tests
>>
>> flink-contrib
>>   -> flink-storm-compatibility
>>   -> flink-storm-compatibility-examples
>>   -> flink-streaming-utils
>>   -> flink-tweet-inputformat
>>   -> flink-operator-stats
>>   -> flink-tez
>>
>> flink-quickstart
>>   -> flink-quickstart-java
>>   -> flink-quickstart-scala
>>   -> flink-tez-quickstart
>>
>> flink-yarn
>> flink-yarn-tests
>>
>> flink-dist
>>
>> flink-benchmark
>>
>>
>> Let me know if that makes sense!
>>
>> Greetings,
>> Stephan


signature.asc (836 bytes) Download Attachment
mxm
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

mxm
+1 Matthias, let's limit the overhead this has for the module maintainers.

On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]> wrote:

> I will commit something to flink-storm-compatibility tomorrow that
> contains some internal package restructuring. I think, renaming the
> three modules in this commit would be a smart move as both changes
> result in merge conflicts when rebasing open PRs. Thus we can limit this
> pain to a single time. If no objections, I will commit those changes
> tomorrow.
>
> -Matthias
>
> On 10/01/2015 09:52 PM, Henry Saputra wrote:
>> +1
>>
>> I like the idea moving "staging" projects into appropriate modules.
>>
>> While we are at it, I would like to propose changing "
>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
>> but would be nice if it is part of re-org.
>> Supporting Hadoop in the connector implicitly means compatibility with Hadoop.
>> Also same thing with "flink-storm-compatibility" to "flink-storm".
>>
>> - Henry
>>
>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]> wrote:
>>> Hi all!
>>>
>>> We are making good headway with reworking the last parts of the Window API.
>>> After that, the streaming API should be good to be pulled out of staging.
>>>
>>> Since we are reorganizing the projects as part of that, I would shift a bit
>>> more to bring things a bit more up to date.
>>>
>>> In this restructure, I would like to get rid of the "flink-staging"
>>> project. Anyone who only uses the maven artifacts sees no difference
>>> whether a project is in "staging" or not, so it does not help much to have
>>> that directory structure.
>>> On the other hand, projects have a tendency to linger in staging forever
>>> (like avro, spargel, hbase, jdbc, ...)
>>>
>>> The new structure could be
>>>
>>> flink-core
>>> flink-java
>>> flink-scala
>>> flink-streaming-core
>>> flink-streaming-scala
>>>
>>> flink-runtime
>>> flink-runtime-web
>>> flink-optimizer
>>> flink-clients
>>>
>>> flink-shaded
>>>   -> flink-shaded-hadoop
>>>   -> flink-shaded-hadoop2
>>>   -> flink-shaded-include-yarn-tests
>>>   -> flink-shaded-curator
>>>
>>> flink-examples
>>>   -> (have all examples, Scala and Java, Batch and Streaming)
>>>
>>> flink-batch-connectors
>>>   -> flink-avro
>>>   -> flink-jdbc
>>>   -> flink-hadoop-compatibility
>>>   -> flink-hbase
>>>   -> flink-hcatalog
>>>
>>> flink-streaming-connectors
>>>   -> flink-connector-twitter
>>>   -> flink-streaming-examples
>>>   -> flink-connector-flume
>>>   -> flink-connector-kafka
>>>   -> flink-connector-elasticsearch
>>>   -> flink-connector-rabbitmq
>>>   -> flink-connector-filesystem
>>>
>>> flink-libraries
>>>   -> flink-gelly
>>>   -> flink-gelly-scala
>>>   -> flink-ml
>>>   -> flink-table
>>>   -> flink-language-binding
>>>   -> flink-python
>>>
>>>
>>> flink-scala-shell
>>>
>>> flink-test-utils
>>> flink-tests
>>> flink-fs-tests
>>>
>>> flink-contrib
>>>   -> flink-storm-compatibility
>>>   -> flink-storm-compatibility-examples
>>>   -> flink-streaming-utils
>>>   -> flink-tweet-inputformat
>>>   -> flink-operator-stats
>>>   -> flink-tez
>>>
>>> flink-quickstart
>>>   -> flink-quickstart-java
>>>   -> flink-quickstart-scala
>>>   -> flink-tez-quickstart
>>>
>>> flink-yarn
>>> flink-yarn-tests
>>>
>>> flink-dist
>>>
>>> flink-benchmark
>>>
>>>
>>> Let me know if that makes sense!
>>>
>>> Greetings,
>>> Stephan
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Till Rohrmann
+1 for the new project structure. Getting rid of our code dump is a good
thing.

On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]> wrote:

> +1 Matthias, let's limit the overhead this has for the module maintainers.
>
> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]> wrote:
> > I will commit something to flink-storm-compatibility tomorrow that
> > contains some internal package restructuring. I think, renaming the
> > three modules in this commit would be a smart move as both changes
> > result in merge conflicts when rebasing open PRs. Thus we can limit this
> > pain to a single time. If no objections, I will commit those changes
> > tomorrow.
> >
> > -Matthias
> >
> > On 10/01/2015 09:52 PM, Henry Saputra wrote:
> >> +1
> >>
> >> I like the idea moving "staging" projects into appropriate modules.
> >>
> >> While we are at it, I would like to propose changing "
> >> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
> >> but would be nice if it is part of re-org.
> >> Supporting Hadoop in the connector implicitly means compatibility with
> Hadoop.
> >> Also same thing with "flink-storm-compatibility" to "flink-storm".
> >>
> >> - Henry
> >>
> >> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]> wrote:
> >>> Hi all!
> >>>
> >>> We are making good headway with reworking the last parts of the Window
> API.
> >>> After that, the streaming API should be good to be pulled out of
> staging.
> >>>
> >>> Since we are reorganizing the projects as part of that, I would shift
> a bit
> >>> more to bring things a bit more up to date.
> >>>
> >>> In this restructure, I would like to get rid of the "flink-staging"
> >>> project. Anyone who only uses the maven artifacts sees no difference
> >>> whether a project is in "staging" or not, so it does not help much to
> have
> >>> that directory structure.
> >>> On the other hand, projects have a tendency to linger in staging
> forever
> >>> (like avro, spargel, hbase, jdbc, ...)
> >>>
> >>> The new structure could be
> >>>
> >>> flink-core
> >>> flink-java
> >>> flink-scala
> >>> flink-streaming-core
> >>> flink-streaming-scala
> >>>
> >>> flink-runtime
> >>> flink-runtime-web
> >>> flink-optimizer
> >>> flink-clients
> >>>
> >>> flink-shaded
> >>>   -> flink-shaded-hadoop
> >>>   -> flink-shaded-hadoop2
> >>>   -> flink-shaded-include-yarn-tests
> >>>   -> flink-shaded-curator
> >>>
> >>> flink-examples
> >>>   -> (have all examples, Scala and Java, Batch and Streaming)
> >>>
> >>> flink-batch-connectors
> >>>   -> flink-avro
> >>>   -> flink-jdbc
> >>>   -> flink-hadoop-compatibility
> >>>   -> flink-hbase
> >>>   -> flink-hcatalog
> >>>
> >>> flink-streaming-connectors
> >>>   -> flink-connector-twitter
> >>>   -> flink-streaming-examples
> >>>   -> flink-connector-flume
> >>>   -> flink-connector-kafka
> >>>   -> flink-connector-elasticsearch
> >>>   -> flink-connector-rabbitmq
> >>>   -> flink-connector-filesystem
> >>>
> >>> flink-libraries
> >>>   -> flink-gelly
> >>>   -> flink-gelly-scala
> >>>   -> flink-ml
> >>>   -> flink-table
> >>>   -> flink-language-binding
> >>>   -> flink-python
> >>>
> >>>
> >>> flink-scala-shell
> >>>
> >>> flink-test-utils
> >>> flink-tests
> >>> flink-fs-tests
> >>>
> >>> flink-contrib
> >>>   -> flink-storm-compatibility
> >>>   -> flink-storm-compatibility-examples
> >>>   -> flink-streaming-utils
> >>>   -> flink-tweet-inputformat
> >>>   -> flink-operator-stats
> >>>   -> flink-tez
> >>>
> >>> flink-quickstart
> >>>   -> flink-quickstart-java
> >>>   -> flink-quickstart-scala
> >>>   -> flink-tez-quickstart
> >>>
> >>> flink-yarn
> >>> flink-yarn-tests
> >>>
> >>> flink-dist
> >>>
> >>> flink-benchmark
> >>>
> >>>
> >>> Let me know if that makes sense!
> >>>
> >>> Greetings,
> >>> Stephan
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Stephan Ewen
@Matthias: How about getting rid of the storm-compatibility-parent and
making the core and examples projects directly projects in "contrib"

On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]> wrote:

> +1 for the new project structure. Getting rid of our code dump is a good
> thing.
>
> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]>
> wrote:
>
> > +1 Matthias, let's limit the overhead this has for the module
> maintainers.
> >
> > On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]>
> wrote:
> > > I will commit something to flink-storm-compatibility tomorrow that
> > > contains some internal package restructuring. I think, renaming the
> > > three modules in this commit would be a smart move as both changes
> > > result in merge conflicts when rebasing open PRs. Thus we can limit
> this
> > > pain to a single time. If no objections, I will commit those changes
> > > tomorrow.
> > >
> > > -Matthias
> > >
> > > On 10/01/2015 09:52 PM, Henry Saputra wrote:
> > >> +1
> > >>
> > >> I like the idea moving "staging" projects into appropriate modules.
> > >>
> > >> While we are at it, I would like to propose changing "
> > >> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
> > >> but would be nice if it is part of re-org.
> > >> Supporting Hadoop in the connector implicitly means compatibility with
> > Hadoop.
> > >> Also same thing with "flink-storm-compatibility" to "flink-storm".
> > >>
> > >> - Henry
> > >>
> > >> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]>
> wrote:
> > >>> Hi all!
> > >>>
> > >>> We are making good headway with reworking the last parts of the
> Window
> > API.
> > >>> After that, the streaming API should be good to be pulled out of
> > staging.
> > >>>
> > >>> Since we are reorganizing the projects as part of that, I would shift
> > a bit
> > >>> more to bring things a bit more up to date.
> > >>>
> > >>> In this restructure, I would like to get rid of the "flink-staging"
> > >>> project. Anyone who only uses the maven artifacts sees no difference
> > >>> whether a project is in "staging" or not, so it does not help much to
> > have
> > >>> that directory structure.
> > >>> On the other hand, projects have a tendency to linger in staging
> > forever
> > >>> (like avro, spargel, hbase, jdbc, ...)
> > >>>
> > >>> The new structure could be
> > >>>
> > >>> flink-core
> > >>> flink-java
> > >>> flink-scala
> > >>> flink-streaming-core
> > >>> flink-streaming-scala
> > >>>
> > >>> flink-runtime
> > >>> flink-runtime-web
> > >>> flink-optimizer
> > >>> flink-clients
> > >>>
> > >>> flink-shaded
> > >>>   -> flink-shaded-hadoop
> > >>>   -> flink-shaded-hadoop2
> > >>>   -> flink-shaded-include-yarn-tests
> > >>>   -> flink-shaded-curator
> > >>>
> > >>> flink-examples
> > >>>   -> (have all examples, Scala and Java, Batch and Streaming)
> > >>>
> > >>> flink-batch-connectors
> > >>>   -> flink-avro
> > >>>   -> flink-jdbc
> > >>>   -> flink-hadoop-compatibility
> > >>>   -> flink-hbase
> > >>>   -> flink-hcatalog
> > >>>
> > >>> flink-streaming-connectors
> > >>>   -> flink-connector-twitter
> > >>>   -> flink-streaming-examples
> > >>>   -> flink-connector-flume
> > >>>   -> flink-connector-kafka
> > >>>   -> flink-connector-elasticsearch
> > >>>   -> flink-connector-rabbitmq
> > >>>   -> flink-connector-filesystem
> > >>>
> > >>> flink-libraries
> > >>>   -> flink-gelly
> > >>>   -> flink-gelly-scala
> > >>>   -> flink-ml
> > >>>   -> flink-table
> > >>>   -> flink-language-binding
> > >>>   -> flink-python
> > >>>
> > >>>
> > >>> flink-scala-shell
> > >>>
> > >>> flink-test-utils
> > >>> flink-tests
> > >>> flink-fs-tests
> > >>>
> > >>> flink-contrib
> > >>>   -> flink-storm-compatibility
> > >>>   -> flink-storm-compatibility-examples
> > >>>   -> flink-streaming-utils
> > >>>   -> flink-tweet-inputformat
> > >>>   -> flink-operator-stats
> > >>>   -> flink-tez
> > >>>
> > >>> flink-quickstart
> > >>>   -> flink-quickstart-java
> > >>>   -> flink-quickstart-scala
> > >>>   -> flink-tez-quickstart
> > >>>
> > >>> flink-yarn
> > >>> flink-yarn-tests
> > >>>
> > >>> flink-dist
> > >>>
> > >>> flink-benchmark
> > >>>
> > >>>
> > >>> Let me know if that makes sense!
> > >>>
> > >>> Greetings,
> > >>> Stephan
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Matthias J. Sax-2
Sure. Will do that.

-Matthias

On 10/02/2015 10:35 AM, Stephan Ewen wrote:

> @Matthias: How about getting rid of the storm-compatibility-parent and
> making the core and examples projects directly projects in "contrib"
>
> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]> wrote:
>
>> +1 for the new project structure. Getting rid of our code dump is a good
>> thing.
>>
>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]>
>> wrote:
>>
>>> +1 Matthias, let's limit the overhead this has for the module
>> maintainers.
>>>
>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]>
>> wrote:
>>>> I will commit something to flink-storm-compatibility tomorrow that
>>>> contains some internal package restructuring. I think, renaming the
>>>> three modules in this commit would be a smart move as both changes
>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
>> this
>>>> pain to a single time. If no objections, I will commit those changes
>>>> tomorrow.
>>>>
>>>> -Matthias
>>>>
>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
>>>>> +1
>>>>>
>>>>> I like the idea moving "staging" projects into appropriate modules.
>>>>>
>>>>> While we are at it, I would like to propose changing "
>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
>>>>> but would be nice if it is part of re-org.
>>>>> Supporting Hadoop in the connector implicitly means compatibility with
>>> Hadoop.
>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
>>>>>
>>>>> - Henry
>>>>>
>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]>
>> wrote:
>>>>>> Hi all!
>>>>>>
>>>>>> We are making good headway with reworking the last parts of the
>> Window
>>> API.
>>>>>> After that, the streaming API should be good to be pulled out of
>>> staging.
>>>>>>
>>>>>> Since we are reorganizing the projects as part of that, I would shift
>>> a bit
>>>>>> more to bring things a bit more up to date.
>>>>>>
>>>>>> In this restructure, I would like to get rid of the "flink-staging"
>>>>>> project. Anyone who only uses the maven artifacts sees no difference
>>>>>> whether a project is in "staging" or not, so it does not help much to
>>> have
>>>>>> that directory structure.
>>>>>> On the other hand, projects have a tendency to linger in staging
>>> forever
>>>>>> (like avro, spargel, hbase, jdbc, ...)
>>>>>>
>>>>>> The new structure could be
>>>>>>
>>>>>> flink-core
>>>>>> flink-java
>>>>>> flink-scala
>>>>>> flink-streaming-core
>>>>>> flink-streaming-scala
>>>>>>
>>>>>> flink-runtime
>>>>>> flink-runtime-web
>>>>>> flink-optimizer
>>>>>> flink-clients
>>>>>>
>>>>>> flink-shaded
>>>>>>   -> flink-shaded-hadoop
>>>>>>   -> flink-shaded-hadoop2
>>>>>>   -> flink-shaded-include-yarn-tests
>>>>>>   -> flink-shaded-curator
>>>>>>
>>>>>> flink-examples
>>>>>>   -> (have all examples, Scala and Java, Batch and Streaming)
>>>>>>
>>>>>> flink-batch-connectors
>>>>>>   -> flink-avro
>>>>>>   -> flink-jdbc
>>>>>>   -> flink-hadoop-compatibility
>>>>>>   -> flink-hbase
>>>>>>   -> flink-hcatalog
>>>>>>
>>>>>> flink-streaming-connectors
>>>>>>   -> flink-connector-twitter
>>>>>>   -> flink-streaming-examples
>>>>>>   -> flink-connector-flume
>>>>>>   -> flink-connector-kafka
>>>>>>   -> flink-connector-elasticsearch
>>>>>>   -> flink-connector-rabbitmq
>>>>>>   -> flink-connector-filesystem
>>>>>>
>>>>>> flink-libraries
>>>>>>   -> flink-gelly
>>>>>>   -> flink-gelly-scala
>>>>>>   -> flink-ml
>>>>>>   -> flink-table
>>>>>>   -> flink-language-binding
>>>>>>   -> flink-python
>>>>>>
>>>>>>
>>>>>> flink-scala-shell
>>>>>>
>>>>>> flink-test-utils
>>>>>> flink-tests
>>>>>> flink-fs-tests
>>>>>>
>>>>>> flink-contrib
>>>>>>   -> flink-storm-compatibility
>>>>>>   -> flink-storm-compatibility-examples
>>>>>>   -> flink-streaming-utils
>>>>>>   -> flink-tweet-inputformat
>>>>>>   -> flink-operator-stats
>>>>>>   -> flink-tez
>>>>>>
>>>>>> flink-quickstart
>>>>>>   -> flink-quickstart-java
>>>>>>   -> flink-quickstart-scala
>>>>>>   -> flink-tez-quickstart
>>>>>>
>>>>>> flink-yarn
>>>>>> flink-yarn-tests
>>>>>>
>>>>>> flink-dist
>>>>>>
>>>>>> flink-benchmark
>>>>>>
>>>>>>
>>>>>> Let me know if that makes sense!
>>>>>>
>>>>>> Greetings,
>>>>>> Stephan
>>>>
>>>
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Matthias J. Sax-2
It think, rename "flink-storm-compatibility-core" to just "flink-storm"
would be the cleanest solution.

So in flink-contrib there would be two modules:
  - flink-storm
  - flink-storm-examples

Please let me know if you have any objection about it.

-Matthias

On 10/02/2015 10:45 AM, Matthias J. Sax wrote:

> Sure. Will do that.
>
> -Matthias
>
> On 10/02/2015 10:35 AM, Stephan Ewen wrote:
>> @Matthias: How about getting rid of the storm-compatibility-parent and
>> making the core and examples projects directly projects in "contrib"
>>
>> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]> wrote:
>>
>>> +1 for the new project structure. Getting rid of our code dump is a good
>>> thing.
>>>
>>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]>
>>> wrote:
>>>
>>>> +1 Matthias, let's limit the overhead this has for the module
>>> maintainers.
>>>>
>>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]>
>>> wrote:
>>>>> I will commit something to flink-storm-compatibility tomorrow that
>>>>> contains some internal package restructuring. I think, renaming the
>>>>> three modules in this commit would be a smart move as both changes
>>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
>>> this
>>>>> pain to a single time. If no objections, I will commit those changes
>>>>> tomorrow.
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
>>>>>> +1
>>>>>>
>>>>>> I like the idea moving "staging" projects into appropriate modules.
>>>>>>
>>>>>> While we are at it, I would like to propose changing "
>>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket list
>>>>>> but would be nice if it is part of re-org.
>>>>>> Supporting Hadoop in the connector implicitly means compatibility with
>>>> Hadoop.
>>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
>>>>>>
>>>>>> - Henry
>>>>>>
>>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]>
>>> wrote:
>>>>>>> Hi all!
>>>>>>>
>>>>>>> We are making good headway with reworking the last parts of the
>>> Window
>>>> API.
>>>>>>> After that, the streaming API should be good to be pulled out of
>>>> staging.
>>>>>>>
>>>>>>> Since we are reorganizing the projects as part of that, I would shift
>>>> a bit
>>>>>>> more to bring things a bit more up to date.
>>>>>>>
>>>>>>> In this restructure, I would like to get rid of the "flink-staging"
>>>>>>> project. Anyone who only uses the maven artifacts sees no difference
>>>>>>> whether a project is in "staging" or not, so it does not help much to
>>>> have
>>>>>>> that directory structure.
>>>>>>> On the other hand, projects have a tendency to linger in staging
>>>> forever
>>>>>>> (like avro, spargel, hbase, jdbc, ...)
>>>>>>>
>>>>>>> The new structure could be
>>>>>>>
>>>>>>> flink-core
>>>>>>> flink-java
>>>>>>> flink-scala
>>>>>>> flink-streaming-core
>>>>>>> flink-streaming-scala
>>>>>>>
>>>>>>> flink-runtime
>>>>>>> flink-runtime-web
>>>>>>> flink-optimizer
>>>>>>> flink-clients
>>>>>>>
>>>>>>> flink-shaded
>>>>>>>   -> flink-shaded-hadoop
>>>>>>>   -> flink-shaded-hadoop2
>>>>>>>   -> flink-shaded-include-yarn-tests
>>>>>>>   -> flink-shaded-curator
>>>>>>>
>>>>>>> flink-examples
>>>>>>>   -> (have all examples, Scala and Java, Batch and Streaming)
>>>>>>>
>>>>>>> flink-batch-connectors
>>>>>>>   -> flink-avro
>>>>>>>   -> flink-jdbc
>>>>>>>   -> flink-hadoop-compatibility
>>>>>>>   -> flink-hbase
>>>>>>>   -> flink-hcatalog
>>>>>>>
>>>>>>> flink-streaming-connectors
>>>>>>>   -> flink-connector-twitter
>>>>>>>   -> flink-streaming-examples
>>>>>>>   -> flink-connector-flume
>>>>>>>   -> flink-connector-kafka
>>>>>>>   -> flink-connector-elasticsearch
>>>>>>>   -> flink-connector-rabbitmq
>>>>>>>   -> flink-connector-filesystem
>>>>>>>
>>>>>>> flink-libraries
>>>>>>>   -> flink-gelly
>>>>>>>   -> flink-gelly-scala
>>>>>>>   -> flink-ml
>>>>>>>   -> flink-table
>>>>>>>   -> flink-language-binding
>>>>>>>   -> flink-python
>>>>>>>
>>>>>>>
>>>>>>> flink-scala-shell
>>>>>>>
>>>>>>> flink-test-utils
>>>>>>> flink-tests
>>>>>>> flink-fs-tests
>>>>>>>
>>>>>>> flink-contrib
>>>>>>>   -> flink-storm-compatibility
>>>>>>>   -> flink-storm-compatibility-examples
>>>>>>>   -> flink-streaming-utils
>>>>>>>   -> flink-tweet-inputformat
>>>>>>>   -> flink-operator-stats
>>>>>>>   -> flink-tez
>>>>>>>
>>>>>>> flink-quickstart
>>>>>>>   -> flink-quickstart-java
>>>>>>>   -> flink-quickstart-scala
>>>>>>>   -> flink-tez-quickstart
>>>>>>>
>>>>>>> flink-yarn
>>>>>>> flink-yarn-tests
>>>>>>>
>>>>>>> flink-dist
>>>>>>>
>>>>>>> flink-benchmark
>>>>>>>
>>>>>>>
>>>>>>> Let me know if that makes sense!
>>>>>>>
>>>>>>> Greetings,
>>>>>>> Stephan
>>>>>
>>>>
>>>
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Pulling Streaming out of staging and project restructure

Stephan Ewen
@matthias +1 for that approach

On Fri, Oct 2, 2015 at 11:21 AM, Matthias J. Sax <[hidden email]> wrote:

> It think, rename "flink-storm-compatibility-core" to just "flink-storm"
> would be the cleanest solution.
>
> So in flink-contrib there would be two modules:
>   - flink-storm
>   - flink-storm-examples
>
> Please let me know if you have any objection about it.
>
> -Matthias
>
> On 10/02/2015 10:45 AM, Matthias J. Sax wrote:
> > Sure. Will do that.
> >
> > -Matthias
> >
> > On 10/02/2015 10:35 AM, Stephan Ewen wrote:
> >> @Matthias: How about getting rid of the storm-compatibility-parent and
> >> making the core and examples projects directly projects in "contrib"
> >>
> >> On Fri, Oct 2, 2015 at 10:34 AM, Till Rohrmann <[hidden email]>
> wrote:
> >>
> >>> +1 for the new project structure. Getting rid of our code dump is a
> good
> >>> thing.
> >>>
> >>> On Fri, Oct 2, 2015 at 10:25 AM, Maximilian Michels <[hidden email]>
> >>> wrote:
> >>>
> >>>> +1 Matthias, let's limit the overhead this has for the module
> >>> maintainers.
> >>>>
> >>>> On Fri, Oct 2, 2015 at 12:17 AM, Matthias J. Sax <[hidden email]>
> >>> wrote:
> >>>>> I will commit something to flink-storm-compatibility tomorrow that
> >>>>> contains some internal package restructuring. I think, renaming the
> >>>>> three modules in this commit would be a smart move as both changes
> >>>>> result in merge conflicts when rebasing open PRs. Thus we can limit
> >>> this
> >>>>> pain to a single time. If no objections, I will commit those changes
> >>>>> tomorrow.
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 10/01/2015 09:52 PM, Henry Saputra wrote:
> >>>>>> +1
> >>>>>>
> >>>>>> I like the idea moving "staging" projects into appropriate modules.
> >>>>>>
> >>>>>> While we are at it, I would like to propose changing "
> >>>>>> flink-hadoop-compatibility" to "flink-hadoop". It is in my bucket
> list
> >>>>>> but would be nice if it is part of re-org.
> >>>>>> Supporting Hadoop in the connector implicitly means compatibility
> with
> >>>> Hadoop.
> >>>>>> Also same thing with "flink-storm-compatibility" to "flink-storm".
> >>>>>>
> >>>>>> - Henry
> >>>>>>
> >>>>>> On Thu, Oct 1, 2015 at 3:25 AM, Stephan Ewen <[hidden email]>
> >>> wrote:
> >>>>>>> Hi all!
> >>>>>>>
> >>>>>>> We are making good headway with reworking the last parts of the
> >>> Window
> >>>> API.
> >>>>>>> After that, the streaming API should be good to be pulled out of
> >>>> staging.
> >>>>>>>
> >>>>>>> Since we are reorganizing the projects as part of that, I would
> shift
> >>>> a bit
> >>>>>>> more to bring things a bit more up to date.
> >>>>>>>
> >>>>>>> In this restructure, I would like to get rid of the "flink-staging"
> >>>>>>> project. Anyone who only uses the maven artifacts sees no
> difference
> >>>>>>> whether a project is in "staging" or not, so it does not help much
> to
> >>>> have
> >>>>>>> that directory structure.
> >>>>>>> On the other hand, projects have a tendency to linger in staging
> >>>> forever
> >>>>>>> (like avro, spargel, hbase, jdbc, ...)
> >>>>>>>
> >>>>>>> The new structure could be
> >>>>>>>
> >>>>>>> flink-core
> >>>>>>> flink-java
> >>>>>>> flink-scala
> >>>>>>> flink-streaming-core
> >>>>>>> flink-streaming-scala
> >>>>>>>
> >>>>>>> flink-runtime
> >>>>>>> flink-runtime-web
> >>>>>>> flink-optimizer
> >>>>>>> flink-clients
> >>>>>>>
> >>>>>>> flink-shaded
> >>>>>>>   -> flink-shaded-hadoop
> >>>>>>>   -> flink-shaded-hadoop2
> >>>>>>>   -> flink-shaded-include-yarn-tests
> >>>>>>>   -> flink-shaded-curator
> >>>>>>>
> >>>>>>> flink-examples
> >>>>>>>   -> (have all examples, Scala and Java, Batch and Streaming)
> >>>>>>>
> >>>>>>> flink-batch-connectors
> >>>>>>>   -> flink-avro
> >>>>>>>   -> flink-jdbc
> >>>>>>>   -> flink-hadoop-compatibility
> >>>>>>>   -> flink-hbase
> >>>>>>>   -> flink-hcatalog
> >>>>>>>
> >>>>>>> flink-streaming-connectors
> >>>>>>>   -> flink-connector-twitter
> >>>>>>>   -> flink-streaming-examples
> >>>>>>>   -> flink-connector-flume
> >>>>>>>   -> flink-connector-kafka
> >>>>>>>   -> flink-connector-elasticsearch
> >>>>>>>   -> flink-connector-rabbitmq
> >>>>>>>   -> flink-connector-filesystem
> >>>>>>>
> >>>>>>> flink-libraries
> >>>>>>>   -> flink-gelly
> >>>>>>>   -> flink-gelly-scala
> >>>>>>>   -> flink-ml
> >>>>>>>   -> flink-table
> >>>>>>>   -> flink-language-binding
> >>>>>>>   -> flink-python
> >>>>>>>
> >>>>>>>
> >>>>>>> flink-scala-shell
> >>>>>>>
> >>>>>>> flink-test-utils
> >>>>>>> flink-tests
> >>>>>>> flink-fs-tests
> >>>>>>>
> >>>>>>> flink-contrib
> >>>>>>>   -> flink-storm-compatibility
> >>>>>>>   -> flink-storm-compatibility-examples
> >>>>>>>   -> flink-streaming-utils
> >>>>>>>   -> flink-tweet-inputformat
> >>>>>>>   -> flink-operator-stats
> >>>>>>>   -> flink-tez
> >>>>>>>
> >>>>>>> flink-quickstart
> >>>>>>>   -> flink-quickstart-java
> >>>>>>>   -> flink-quickstart-scala
> >>>>>>>   -> flink-tez-quickstart
> >>>>>>>
> >>>>>>> flink-yarn
> >>>>>>> flink-yarn-tests
> >>>>>>>
> >>>>>>> flink-dist
> >>>>>>>
> >>>>>>> flink-benchmark
> >>>>>>>
> >>>>>>>
> >>>>>>> Let me know if that makes sense!
> >>>>>>>
> >>>>>>> Greetings,
> >>>>>>> Stephan
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
12