Scala 2.10/2.11 Maven dependencies

[mxm]

+1 for a big notice once we merge this.

I would like to have a suffix-free "flink-streaming-java". However,
I'm having a hard time to refactor the streaming-java code to get rid
of Scala. The streaming API depends on "flink-clients" and
"flink-runtime" which both inherently depend on Scala. Unfortunately,
the streaming API is very much chained with the runtime which makes it
hard to move classes from flink-streaming-java to flink-runtime.

I think the best solution would be to have similar abstraction for the
streaming API as we have in batch. That is, to generate a Plan
independently of the runtime which is later turned into a JobGraph. In
streaming, we have the StreamGraph but it is not runtime independent.
It's a bit of a mess.

Another option I see is to shade away Scala. I did this just to see
how large the binaries would be. It would let "flink-streaming-java"
grow from 3,1 MB (without shading) to 59,7 MB (with shading). That
seems a bit too large for users although it would get rid of the Scala
suffix without refactoring.

On Thu, Jan 21, 2016 at 11:33 PM, Ufuk Celebi <[hidden email]> wrote:

>
>> On 21 Jan 2016, at 17:51, Maximilian Michels <[hidden email]> wrote:
>>
>> https://issues.apache.org/jira/browse/FLINK-2940
>>
>> There is now a pending pull request: https://github.com/apache/flink/pull/1529
>>
>> As I was working on the changes, I discovered we have some more
>> modules which have a Scala dependency which could be avoided. Namely
>> these are "flink-java8", and "flink-streaming-java". The latter
>> probably affects a lot of users. So it would be nice to make it
>> Scala-free.
>>
>> Do you think that would be feasible?
>
> Thank you very much for taking care of this. :)
>
> I would like to have it changed as well before merging for the same reason Fabian brought up in the PR.
>
> Since this will be quite a breaking change, we should consider adding a big notice to the 1.0-SNAPSHOT docs regarding this (basically on the landing page) for the few weeks until the release.
>
> – Ufuk
>

[mxm]

I won't have the time to finish the refactoring. Also, it will be
pretty painful with all the large streaming pull requests being merged
at the moment. If there are no objections, I would like to merge the
Scala suffix changes with "flink-streaming-java" being Scala
dependent. It will improve the experience for users in the long run.

After merging, I'll announce the new Scala suffixed modules. In
addition, we could deploy an empty Maven module to overwrite the
current snapshot version of flink-streaming-java. That would prevent
conflicts with different snapshot versions, e.g. combining
flink-streaming-java (instead of flink-streaming-java_2.11) with
flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
problem anymore.

Any objections?

On Fri, Jan 22, 2016 at 6:30 PM, Maximilian Michels <[hidden email]> wrote:

> +1 for a big notice once we merge this.
>
> I would like to have a suffix-free "flink-streaming-java". However,
> I'm having a hard time to refactor the streaming-java code to get rid
> of Scala. The streaming API depends on "flink-clients" and
> "flink-runtime" which both inherently depend on Scala. Unfortunately,
> the streaming API is very much chained with the runtime which makes it
> hard to move classes from flink-streaming-java to flink-runtime.
>
> I think the best solution would be to have similar abstraction for the
> streaming API as we have in batch. That is, to generate a Plan
> independently of the runtime which is later turned into a JobGraph. In
> streaming, we have the StreamGraph but it is not runtime independent.
> It's a bit of a mess.
>
> Another option I see is to shade away Scala. I did this just to see
> how large the binaries would be. It would let "flink-streaming-java"
> grow from 3,1 MB (without shading) to 59,7 MB (with shading). That
> seems a bit too large for users although it would get rid of the Scala
> suffix without refactoring.
>
> On Thu, Jan 21, 2016 at 11:33 PM, Ufuk Celebi <[hidden email]> wrote:
>>
>>> On 21 Jan 2016, at 17:51, Maximilian Michels <[hidden email]> wrote:
>>>
>>> https://issues.apache.org/jira/browse/FLINK-2940
>>>
>>> There is now a pending pull request: https://github.com/apache/flink/pull/1529
>>>
>>> As I was working on the changes, I discovered we have some more
>>> modules which have a Scala dependency which could be avoided. Namely
>>> these are "flink-java8", and "flink-streaming-java". The latter
>>> probably affects a lot of users. So it would be nice to make it
>>> Scala-free.
>>>
>>> Do you think that would be feasible?
>>
>> Thank you very much for taking care of this. :)
>>
>> I would like to have it changed as well before merging for the same reason Fabian brought up in the PR.
>>
>> Since this will be quite a breaking change, we should consider adding a big notice to the 1.0-SNAPSHOT docs regarding this (basically on the landing page) for the few weeks until the release.
>>
>> – Ufuk
>>

[Ufuk Celebi-2]

> On 25 Jan 2016, at 11:39, Maximilian Michels <[hidden email]> wrote:
>
> I won't have the time to finish the refactoring. Also, it will be
> pretty painful with all the large streaming pull requests being merged
> at the moment. If there are no objections, I would like to merge the
> Scala suffix changes with "flink-streaming-java" being Scala
> dependent. It will improve the experience for users in the long run.
>
> After merging, I'll announce the new Scala suffixed modules. In
> addition, we could deploy an empty Maven module to overwrite the
> current snapshot version of flink-streaming-java. That would prevent
> conflicts with different snapshot versions, e.g. combining
> flink-streaming-java (instead of flink-streaming-java_2.11) with
> flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
> problem anymore.
>
> Any objections?

Sounds good to me!

– Ufuk

[Stephan Ewen]

+1

On Mon, Jan 25, 2016 at 11:41 AM, Ufuk Celebi <[hidden email]> wrote:

>
> > On 25 Jan 2016, at 11:39, Maximilian Michels <[hidden email]> wrote:
> >
> > I won't have the time to finish the refactoring. Also, it will be
> > pretty painful with all the large streaming pull requests being merged
> > at the moment. If there are no objections, I would like to merge the
> > Scala suffix changes with "flink-streaming-java" being Scala
> > dependent. It will improve the experience for users in the long run.
> >
> > After merging, I'll announce the new Scala suffixed modules. In
> > addition, we could deploy an empty Maven module to overwrite the
> > current snapshot version of flink-streaming-java. That would prevent
> > conflicts with different snapshot versions, e.g. combining
> > flink-streaming-java (instead of flink-streaming-java_2.11) with
> > flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
> > problem anymore.
> >
> > Any objections?
>
> Sounds good to me!
>
> – Ufuk
>
>

[Till Rohrmann]

+1

On Mon, Jan 25, 2016 at 3:36 PM, Stephan Ewen <[hidden email]> wrote:

> +1
>
> On Mon, Jan 25, 2016 at 11:41 AM, Ufuk Celebi <[hidden email]> wrote:
>
> >
> > > On 25 Jan 2016, at 11:39, Maximilian Michels <[hidden email]> wrote:
> > >
> > > I won't have the time to finish the refactoring. Also, it will be
> > > pretty painful with all the large streaming pull requests being merged
> > > at the moment. If there are no objections, I would like to merge the
> > > Scala suffix changes with "flink-streaming-java" being Scala
> > > dependent. It will improve the experience for users in the long run.
> > >
> > > After merging, I'll announce the new Scala suffixed modules. In
> > > addition, we could deploy an empty Maven module to overwrite the
> > > current snapshot version of flink-streaming-java. That would prevent
> > > conflicts with different snapshot versions, e.g. combining
> > > flink-streaming-java (instead of flink-streaming-java_2.11) with
> > > flink-runtime_2.11. Once 1.0.0 is out, the old artifacts won't be a
> > > problem anymore.
> > >
> > > Any objections?
> >
> > Sounds good to me!
> >
> > – Ufuk
> >
> >
>