(DEPRECATED) Apache Flink Mailing List archive.

Project restructure

Classic

List

Threaded

7 messages Options

Márton Balassi-3

Project restructure

During a recent PR of the streaming scala api [1] arose the issue of
possibly changing the project structure. For the discussion it seems to me
that we should address this as a separate issue. Things to note:

* According to Stephan for the batch part, there are discussions
to combine the "flink-core", "flink-java" projects, possibly also the
"flink-scala" project. We are starting to see too many interdependencies.
[2]
* Streaming is currently under flink-addons, but we are positive that
for the next version we can come up with a fairly stable api if needed. We
would like to have it top level eventually.
* Minor issue to keep in mind: Developing our projects with both scala
and java nature seems a bit flaky at the moment at least for Eclipse. [3]
Proposed solutions are also there, just let us make sure to give new
developers a smooth experience with Flink.

I personally like the following suggestion: [2]

We could, in the next version, go for something like
- flink-core (core and batch, java & scala)
- flink-streaming (java & scala)
- flink-runtime
- ...

Ufuk also +1'd this.

As currently the merge of [1] is blocking further streaming development (it
also contains some refactor) I'd like to merge it asap to where it is
currently (flink-scala), and let us figure out the project restructure
separately. Added a JIRA for the latter. [4] If we choose to restructure
the project it will need a commit anyway.

[1] https://github.com/apache/incubator-flink/pull/275
[2] https://github.com/apache/incubator-flink/pull/275#issuecomment-68049822
[3]
http://mail-archives.apache.org/mod_mbox/incubator-flink-dev/201412.mbox/%3CCANC1h_tLtGeOxT-aaA5KR6V4m-Efz8fSN5yKcdX%2B7sjeTdFBEw%40mail.gmail.com%3E
[4] https://issues.apache.org/jira/browse/FLINK-1340

Please when replying vote and comment on the restructure and merge
separately.

Best,

Marton

Márton Balassi-3

Re: Project restructure

Hey,

Any views on this please? We would like to merge as soon as possible.

Thanks,

Marton

On Thu, Dec 25, 2014 at 6:28 PM, Márton Balassi <[hidden email]> wrote:

> During a recent PR of the streaming scala api [1] arose the issue of
> possibly changing the project structure. For the discussion it seems to me
> that we should address this as a separate issue. Things to note:
>
> * According to Stephan for the batch part, there are discussions
> to combine the "flink-core", "flink-java" projects, possibly also the
> "flink-scala" project. We are starting to see too many interdependencies.
> [2]
> * Streaming is currently under flink-addons, but we are positive that
> for the next version we can come up with a fairly stable api if needed. We
> would like to have it top level eventually.
> * Minor issue to keep in mind: Developing our projects with both scala
> and java nature seems a bit flaky at the moment at least for Eclipse. [3]
> Proposed solutions are also there, just let us make sure to give new
> developers a smooth experience with Flink.
>
> I personally like the following suggestion: [2]
>
> We could, in the next version, go for something like
> - flink-core (core and batch, java & scala)
> - flink-streaming (java & scala)
> - flink-runtime
> - ...
>
> Ufuk also +1'd this.
>
> As currently the merge of [1] is blocking further streaming development
> (it also contains some refactor) I'd like to merge it asap to where it is
> currently (flink-scala), and let us figure out the project restructure
> separately. Added a JIRA for the latter. [4] If we choose to restructure
> the project it will need a commit anyway.
>
> [1] https://github.com/apache/incubator-flink/pull/275
> [2]
> https://github.com/apache/incubator-flink/pull/275#issuecomment-68049822
> [3]
> http://mail-archives.apache.org/mod_mbox/incubator-flink-dev/201412.mbox/%3CCANC1h_tLtGeOxT-aaA5KR6V4m-Efz8fSN5yKcdX%2B7sjeTdFBEw%40mail.gmail.com%3E
> [4] https://issues.apache.org/jira/browse/FLINK-1340
>
> Please when replying vote and comment on the restructure and merge
> separately.
>
> Best,
>
> Marton
>
>

Stephan Ewen

Re: Project restructure

Marton,

As far as I understood, this is relevant for the master, not for the 0.8
release, correct?

In that case, I suggest to go ahead with the option you voted for (which is
also supported at least by Ufuk and me as well). If we discover downsides,
we will be able to correct this in the course of the next weeks, well
before the next release.

Greetings,
Stephan

On Sat, Dec 27, 2014 at 8:30 PM, Márton Balassi <[hidden email]> wrote:

> Hey,
>
> Any views on this please? We would like to merge as soon as possible.
>
> Thanks,
>
> Marton
>
> On Thu, Dec 25, 2014 at 6:28 PM, Márton Balassi <[hidden email]>
> wrote:
>
> > During a recent PR of the streaming scala api [1] arose the issue of
> > possibly changing the project structure. For the discussion it seems to
> me
> > that we should address this as a separate issue. Things to note:
> >
> > * According to Stephan for the batch part, there are discussions
> > to combine the "flink-core", "flink-java" projects, possibly also the
> > "flink-scala" project. We are starting to see too many interdependencies.
> > [2]
> > * Streaming is currently under flink-addons, but we are positive that
> > for the next version we can come up with a fairly stable api if needed.
> We
> > would like to have it top level eventually.
> > * Minor issue to keep in mind: Developing our projects with both scala
> > and java nature seems a bit flaky at the moment at least for Eclipse. [3]
> > Proposed solutions are also there, just let us make sure to give new
> > developers a smooth experience with Flink.
> >
> > I personally like the following suggestion: [2]
> >
> > We could, in the next version, go for something like
> > - flink-core (core and batch, java & scala)
> > - flink-streaming (java & scala)
> > - flink-runtime
> > - ...
> >
> > Ufuk also +1'd this.
> >
> > As currently the merge of [1] is blocking further streaming development
> > (it also contains some refactor) I'd like to merge it asap to where it is
> > currently (flink-scala), and let us figure out the project restructure
> > separately. Added a JIRA for the latter. [4] If we choose to restructure
> > the project it will need a commit anyway.
> >
> > [1] https://github.com/apache/incubator-flink/pull/275
> > [2]
> > https://github.com/apache/incubator-flink/pull/275#issuecomment-68049822
> > [3]
> >
> http://mail-archives.apache.org/mod_mbox/incubator-flink-dev/201412.mbox/%3CCANC1h_tLtGeOxT-aaA5KR6V4m-Efz8fSN5yKcdX%2B7sjeTdFBEw%40mail.gmail.com%3E
> > [4] https://issues.apache.org/jira/browse/FLINK-1340
> >
> > Please when replying vote and comment on the restructure and merge
> > separately.
> >
> > Best,
> >
> > Marton
> >
> >
>

Chiwan Park

Re: Project restructure

Hello!

I agree that refactoring project structure is necessary. But there are some pull requests influenced by this change such as #275 [1], #226 [2]. I think that we would better merge these pull requests before refactoring.

[1] https://github.com/apache/incubator-flink/pull/275 <https://github.com/apache/incubator-flink/pull/275>
[2] https://github.com/apache/incubator-flink/pull/226 <https://github.com/apache/incubator-flink/pull/226>

—
Chiwan Park (Sent with iPhone)

> On Dec 28, 2014, at 5:07 AM, Stephan Ewen <[hidden email]> wrote:
>
> Marton,
>
> As far as I understood, this is relevant for the master, not for the 0.8
> release, correct?
>
> In that case, I suggest to go ahead with the option you voted for (which is
> also supported at least by Ufuk and me as well). If we discover downsides,
> we will be able to correct this in the course of the next weeks, well
> before the next release.
>
> Greetings,
> Stephan
>
>
> On Sat, Dec 27, 2014 at 8:30 PM, Márton Balassi <[hidden email]> wrote:
>
>> Hey,
>>
>> Any views on this please? We would like to merge as soon as possible.
>>
>> Thanks,
>>
>> Marton
>>
>> On Thu, Dec 25, 2014 at 6:28 PM, Márton Balassi <[hidden email]>
>> wrote:
>>
>>> During a recent PR of the streaming scala api [1] arose the issue of
>>> possibly changing the project structure. For the discussion it seems to
>> me
>>> that we should address this as a separate issue. Things to note:
>>>
>>> * According to Stephan for the batch part, there are discussions
>>> to combine the "flink-core", "flink-java" projects, possibly also the
>>> "flink-scala" project. We are starting to see too many interdependencies.
>>> [2]
>>> * Streaming is currently under flink-addons, but we are positive that
>>> for the next version we can come up with a fairly stable api if needed.
>> We
>>> would like to have it top level eventually.
>>> * Minor issue to keep in mind: Developing our projects with both scala
>>> and java nature seems a bit flaky at the moment at least for Eclipse. [3]
>>> Proposed solutions are also there, just let us make sure to give new
>>> developers a smooth experience with Flink.
>>>
>>> I personally like the following suggestion: [2]
>>>
>>> We could, in the next version, go for something like
>>> - flink-core (core and batch, java & scala)
>>> - flink-streaming (java & scala)
>>> - flink-runtime
>>> - ...
>>>
>>> Ufuk also +1'd this.
>>>
>>> As currently the merge of [1] is blocking further streaming development
>>> (it also contains some refactor) I'd like to merge it asap to where it is
>>> currently (flink-scala), and let us figure out the project restructure
>>> separately. Added a JIRA for the latter. [4] If we choose to restructure
>>> the project it will need a commit anyway.
>>>
>>> [1] https://github.com/apache/incubator-flink/pull/275
>>> [2]
>>> https://github.com/apache/incubator-flink/pull/275#issuecomment-68049822
>>> [3]
>>>
>> http://mail-archives.apache.org/mod_mbox/incubator-flink-dev/201412.mbox/%3CCANC1h_tLtGeOxT-aaA5KR6V4m-Efz8fSN5yKcdX%2B7sjeTdFBEw%40mail.gmail.com%3E
>>> [4] https://issues.apache.org/jira/browse/FLINK-1340
>>>
>>> Please when replying vote and comment on the restructure and merge
>>> separately.
>>>
>>> Best,
>>>
>>> Marton
>>>
>>>
>>

Fabian Hueske

Re: Project restructure

Hi,

I'm +1 for
- merging the batch API modules
- having streaming top-level when the contributors think it's ready

I have a question though. Why did the streaming Scala API into the batch
Scala API module?
Are there dependency (or other) problems or could it be moved to the other
streaming modules?
Since there are no concrete plans (that I'm aware of) to do the refactoring
soon, I would avoid to have the streaming and batch APIs mixed if there's
not a good reason for that.

Cheers, Fabian

2014-12-28 9:59 GMT+01:00 Chiwan Park <[hidden email]>:

> Hello!
>
> I agree that refactoring project structure is necessary. But there are
> some pull requests influenced by this change such as #275 [1], #226 [2]. I
> think that we would better merge these pull requests before refactoring.
>
> [1] https://github.com/apache/incubator-flink/pull/275 <
> https://github.com/apache/incubator-flink/pull/275>
> [2] https://github.com/apache/incubator-flink/pull/226 <
> https://github.com/apache/incubator-flink/pull/226>
>
> —
> Chiwan Park (Sent with iPhone)
>
>
>
> > On Dec 28, 2014, at 5:07 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > Marton,
> >
> > As far as I understood, this is relevant for the master, not for the 0.8
> > release, correct?
> >
> > In that case, I suggest to go ahead with the option you voted for (which
> is
> > also supported at least by Ufuk and me as well). If we discover
> downsides,
> > we will be able to correct this in the course of the next weeks, well
> > before the next release.
> >
> > Greetings,
> > Stephan
> >
> >
> > On Sat, Dec 27, 2014 at 8:30 PM, Márton Balassi <[hidden email]>
> wrote:
> >
> >> Hey,
> >>
> >> Any views on this please? We would like to merge as soon as possible.
> >>
> >> Thanks,
> >>
> >> Marton
> >>
> >> On Thu, Dec 25, 2014 at 6:28 PM, Márton Balassi <[hidden email]>
> >> wrote:
> >>
> >>> During a recent PR of the streaming scala api [1] arose the issue of
> >>> possibly changing the project structure. For the discussion it seems to
> >> me
> >>> that we should address this as a separate issue. Things to note:
> >>>
> >>> * According to Stephan for the batch part, there are discussions
> >>> to combine the "flink-core", "flink-java" projects, possibly also the
> >>> "flink-scala" project. We are starting to see too many
> interdependencies.
> >>> [2]
> >>> * Streaming is currently under flink-addons, but we are positive that
> >>> for the next version we can come up with a fairly stable api if needed.
> >> We
> >>> would like to have it top level eventually.
> >>> * Minor issue to keep in mind: Developing our projects with both
> scala
> >>> and java nature seems a bit flaky at the moment at least for Eclipse.
> [3]
> >>> Proposed solutions are also there, just let us make sure to give new
> >>> developers a smooth experience with Flink.
> >>>
> >>> I personally like the following suggestion: [2]
> >>>
> >>> We could, in the next version, go for something like
> >>> - flink-core (core and batch, java & scala)
> >>> - flink-streaming (java & scala)
> >>> - flink-runtime
> >>> - ...
> >>>
> >>> Ufuk also +1'd this.
> >>>
> >>> As currently the merge of [1] is blocking further streaming development
> >>> (it also contains some refactor) I'd like to merge it asap to where it
> is
> >>> currently (flink-scala), and let us figure out the project restructure
> >>> separately. Added a JIRA for the latter. [4] If we choose to
> restructure
> >>> the project it will need a commit anyway.
> >>>
> >>> [1] https://github.com/apache/incubator-flink/pull/275
> >>> [2]
> >>>
> https://github.com/apache/incubator-flink/pull/275#issuecomment-68049822
> >>> [3]
> >>>
> >>
> http://mail-archives.apache.org/mod_mbox/incubator-flink-dev/201412.mbox/%3CCANC1h_tLtGeOxT-aaA5KR6V4m-Efz8fSN5yKcdX%2B7sjeTdFBEw%40mail.gmail.com%3E
> >>> [4] https://issues.apache.org/jira/browse/FLINK-1340
> >>>
> >>> Please when replying vote and comment on the restructure and merge
> >>> separately.
> >>>
> >>> Best,
> >>>
> >>> Marton
> >>>
> >>>
> >>
>
>

Márton Balassi

Re: Project restructure

Hey,

@Chiwan: Thanks for the reminder. Sure, I'm sorry that I still haven't had
the time to merge your PR.
@Fabian: The initial idea was that as the scala api is a fairly thin layer
on top of the java thus we could have it in one common project. I've put it
under flink-streaming-scala after I've seen the reactions. :)

As Stephan suggested to have one common examples directory I have not
separated the flink-streaming-examples to java ans scala subprojects. I'm
planning to merge #275 this evening.

On Sun, Dec 28, 2014 at 11:18 AM, Fabian Hueske <[hidden email]> wrote:

> Hi,
>
> I'm +1 for
> - merging the batch API modules
> - having streaming top-level when the contributors think it's ready
>
> I have a question though. Why did the streaming Scala API into the batch
> Scala API module?
> Are there dependency (or other) problems or could it be moved to the other
> streaming modules?
> Since there are no concrete plans (that I'm aware of) to do the refactoring
> soon, I would avoid to have the streaming and batch APIs mixed if there's
> not a good reason for that.
>
> Cheers, Fabian
>
> 2014-12-28 9:59 GMT+01:00 Chiwan Park <[hidden email]>:
>
> > Hello!
> >
> > I agree that refactoring project structure is necessary. But there are
> > some pull requests influenced by this change such as #275 [1], #226 [2].
> I
> > think that we would better merge these pull requests before refactoring.
> >
> > [1] https://github.com/apache/incubator-flink/pull/275 <
> > https://github.com/apache/incubator-flink/pull/275>
> > [2] https://github.com/apache/incubator-flink/pull/226 <
> > https://github.com/apache/incubator-flink/pull/226>
> >
> > —
> > Chiwan Park (Sent with iPhone)
> >
> >
> >
> > > On Dec 28, 2014, at 5:07 AM, Stephan Ewen <[hidden email]> wrote:
> > >
> > > Marton,
> > >
> > > As far as I understood, this is relevant for the master, not for the
> 0.8
> > > release, correct?
> > >
> > > In that case, I suggest to go ahead with the option you voted for
> (which
> > is
> > > also supported at least by Ufuk and me as well). If we discover
> > downsides,
> > > we will be able to correct this in the course of the next weeks, well
> > > before the next release.
> > >
> > > Greetings,
> > > Stephan
> > >
> > >
> > > On Sat, Dec 27, 2014 at 8:30 PM, Márton Balassi <[hidden email]>
> > wrote:
> > >
> > >> Hey,
> > >>
> > >> Any views on this please? We would like to merge as soon as possible.
> > >>
> > >> Thanks,
> > >>
> > >> Marton
> > >>
> > >> On Thu, Dec 25, 2014 at 6:28 PM, Márton Balassi <[hidden email]>
> > >> wrote:
> > >>
> > >>> During a recent PR of the streaming scala api [1] arose the issue of
> > >>> possibly changing the project structure. For the discussion it seems
> to
> > >> me
> > >>> that we should address this as a separate issue. Things to note:
> > >>>
> > >>> * According to Stephan for the batch part, there are discussions
> > >>> to combine the "flink-core", "flink-java" projects, possibly also the
> > >>> "flink-scala" project. We are starting to see too many
> > interdependencies.
> > >>> [2]
> > >>> * Streaming is currently under flink-addons, but we are positive
> that
> > >>> for the next version we can come up with a fairly stable api if
> needed.
> > >> We
> > >>> would like to have it top level eventually.
> > >>> * Minor issue to keep in mind: Developing our projects with both
> > scala
> > >>> and java nature seems a bit flaky at the moment at least for Eclipse.
> > [3]
> > >>> Proposed solutions are also there, just let us make sure to give new
> > >>> developers a smooth experience with Flink.
> > >>>
> > >>> I personally like the following suggestion: [2]
> > >>>
> > >>> We could, in the next version, go for something like
> > >>> - flink-core (core and batch, java & scala)
> > >>> - flink-streaming (java & scala)
> > >>> - flink-runtime
> > >>> - ...
> > >>>
> > >>> Ufuk also +1'd this.
> > >>>
> > >>> As currently the merge of [1] is blocking further streaming
> development
> > >>> (it also contains some refactor) I'd like to merge it asap to where
> it
> > is
> > >>> currently (flink-scala), and let us figure out the project
> restructure
> > >>> separately. Added a JIRA for the latter. [4] If we choose to
> > restructure
> > >>> the project it will need a commit anyway.
> > >>>
> > >>> [1] https://github.com/apache/incubator-flink/pull/275
> > >>> [2]
> > >>>
> > https://github.com/apache/incubator-flink/pull/275#issuecomment-68049822
> > >>> [3]
> > >>>
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/incubator-flink-dev/201412.mbox/%3CCANC1h_tLtGeOxT-aaA5KR6V4m-Efz8fSN5yKcdX%2B7sjeTdFBEw%40mail.gmail.com%3E
> > >>> [4] https://issues.apache.org/jira/browse/FLINK-1340
> > >>>
> > >>> Please when replying vote and comment on the restructure and merge
> > >>> separately.
> > >>>
> > >>> Best,
> > >>>
> > >>> Marton
> > >>>
> > >>>
> > >>
> >
> >
>

Fabian Hueske-2

Re: Project restructure

+1

Thanks Marton!

From: Márton Balassi
Sent: ‎Sunday‎, ‎4‎. ‎January‎, ‎2015 ‎17‎:‎48
To: [hidden email]

Hey,

@Chiwan: Thanks for the reminder. Sure, I'm sorry that I still haven't had
the time to merge your PR.
@Fabian: The initial idea was that as the scala api is a fairly thin layer
on top of the java thus we could have it in one common project. I've put it
under flink-streaming-scala after I've seen the reactions. :)

As Stephan suggested to have one common examples directory I have not
separated the flink-streaming-examples to java ans scala subprojects. I'm
planning to merge #275 this evening.

On Sun, Dec 28, 2014 at 11:18 AM, Fabian Hueske <[hidden email]> wrote: