[jira](FLINK-2493) Simplify names of example program JARs

[Liang Chen]

Summary:Simplify names of example program JARs
Key:FLINK-2493, URL:https://issues.apache.org/jira/browse/FLINK-2493
Issue Type: Improvement
Components:Examples

Propose to simplify the name to "examples/ConnectedComponents.jar" rather than examples/flink-java-examples-0.10-SNAPSHOT-ConnectedComponents.jar

Two proposals, wants to discuss:
Proposal 1:
Change the pom of "flink-java-examples", use executed-example name as "finalName" to replace "default finalName(artifactId+version)+classifier". keep two projects(flink-java-examples,flink-scala-examples) under "flink-examples".

Proposal 2:Combine "flink-java-examples" and "flink-scala-examples" project to one project with "flink-examples", because the Scala example mostly need import  Java example' util/*data*.class, work together.
this would be no any impaction for examples.

please give your ideas, any input would be very appreciated.

[Fabian Hueske-2]

Hi Liang Chen,

thanks for starting this discussion.

I have no strong preference for one of your proposals. Both are fine with
me.

What would proposal 2 mean if we add examples for other API languages, for
example if the Python API moves out of staging?
I guess it shouldn't be a problem, because all examples are self-contained.
Any other thoughts on this?

Best, Fabian

2015-08-14 0:32 GMT+02:00 Liang Chen <[hidden email]>:

> Summary:Simplify names of example program JARs
> Key:FLINK-2493, URL:https://issues.apache.org/jira/browse/FLINK-2493
> Issue Type: Improvement
> Components:Examples
>
> Propose to simplify the name to "examples/ConnectedComponents.jar" rather
> than examples/flink-java-examples-0.10-SNAPSHOT-ConnectedComponents.jar
>
> Two proposals, wants to discuss:
> Proposal 1:
> Change the pom of "flink-java-examples", use executed-example name as
> "finalName" to replace "default finalName(artifactId+version)+classifier".
> keep two projects(flink-java-examples,flink-scala-examples) under
> "flink-examples".
>
> Proposal 2:Combine "flink-java-examples" and "flink-scala-examples" project
> to one project with "flink-examples", because the Scala example mostly need
> import  Java example' util/*data*.class, work together.
> this would be no any impaction for examples.
>
> please give your ideas, any input would be very appreciated.
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/jira-FLINK-2493-Simplify-names-of-example-program-JARs-tp7495.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

[Stephan Ewen]

+1 for proposal (2)

On Fri, Aug 14, 2015 at 10:28 AM, Fabian Hueske <[hidden email]> wrote:

> Hi Liang Chen,
>
> thanks for starting this discussion.
>
> I have no strong preference for one of your proposals. Both are fine with
> me.
>
> What would proposal 2 mean if we add examples for other API languages, for
> example if the Python API moves out of staging?
> I guess it shouldn't be a problem, because all examples are self-contained.
> Any other thoughts on this?
>
> Best, Fabian
>
> 2015-08-14 0:32 GMT+02:00 Liang Chen <[hidden email]>:
>
> > Summary:Simplify names of example program JARs
> > Key:FLINK-2493, URL:https://issues.apache.org/jira/browse/FLINK-2493
> > Issue Type: Improvement
> > Components:Examples
> >
> > Propose to simplify the name to "examples/ConnectedComponents.jar" rather
> > than examples/flink-java-examples-0.10-SNAPSHOT-ConnectedComponents.jar
> >
> > Two proposals, wants to discuss:
> > Proposal 1:
> > Change the pom of "flink-java-examples", use executed-example name as
> > "finalName" to replace "default
> finalName(artifactId+version)+classifier".
> > keep two projects(flink-java-examples,flink-scala-examples) under
> > "flink-examples".
> >
> > Proposal 2:Combine "flink-java-examples" and "flink-scala-examples"
> project
> > to one project with "flink-examples", because the Scala example mostly
> need
> > import  Java example' util/*data*.class, work together.
> > this would be no any impaction for examples.
> >
> > please give your ideas, any input would be very appreciated.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/jira-FLINK-2493-Simplify-names-of-example-program-JARs-tp7495.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>

[till.rohrmann]

In case of merging we would have to add a `scala` tag to some of the
examples because some of the examples exist in both packages. The drawback
of merging both examples modules is that we'll have another mixed
Java/Scala project. This might be bothering for the eclipse folks, but we
already have that. It might make sense to keep the examples separated if
they use different dependencies. Other than that, I think both of your
proposals are fine.

Slightly off-topic: What we could also do is to move the shared components
(e.g. the testing data) to a distinct module (flink-examples/example-data).
But since the generated scala job jars only contain the java data classes
this won't change much at all.

On Fri, Aug 14, 2015 at 10:52 AM, Stephan Ewen <[hidden email]> wrote:

> +1 for proposal (2)
>
> On Fri, Aug 14, 2015 at 10:28 AM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi Liang Chen,
> >
> > thanks for starting this discussion.
> >
> > I have no strong preference for one of your proposals. Both are fine with
> > me.
> >
> > What would proposal 2 mean if we add examples for other API languages,
> for
> > example if the Python API moves out of staging?
> > I guess it shouldn't be a problem, because all examples are
> self-contained.
> > Any other thoughts on this?
> >
> > Best, Fabian
> >
> > 2015-08-14 0:32 GMT+02:00 Liang Chen <[hidden email]>:
> >
> > > Summary:Simplify names of example program JARs
> > > Key:FLINK-2493, URL:https://issues.apache.org/jira/browse/FLINK-2493
> > > Issue Type: Improvement
> > > Components:Examples
> > >
> > > Propose to simplify the name to "examples/ConnectedComponents.jar"
> rather
> > > than examples/flink-java-examples-0.10-SNAPSHOT-ConnectedComponents.jar
> > >
> > > Two proposals, wants to discuss:
> > > Proposal 1:
> > > Change the pom of "flink-java-examples", use executed-example name as
> > > "finalName" to replace "default
> > finalName(artifactId+version)+classifier".
> > > keep two projects(flink-java-examples,flink-scala-examples) under
> > > "flink-examples".
> > >
> > > Proposal 2:Combine "flink-java-examples" and "flink-scala-examples"
> > project
> > > to one project with "flink-examples", because the Scala example mostly
> > need
> > > import  Java example' util/*data*.class, work together.
> > > this would be no any impaction for examples.
> > >
> > > please give your ideas, any input would be very appreciated.
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/jira-FLINK-2493-Simplify-names-of-example-program-JARs-tp7495.html
> > > Sent from the Apache Flink Mailing List archive. mailing list archive
> at
> > > Nabble.com.
> > >
> >
>

[Stephan Ewen]

Concerning mixed projects: Should be okay even for Eclipse users, because
no Scala code is referenced from Java code (that is the tricky part).
Even for someone without a Scala IDE plugin, they would simply not
interpret the "src/main/scala" directory, which should make this appear
like a pure Java project to them.

Concerning name overlap: How about we put them in different packages:
"org.apache.flink.examples.java" and "org.apache.flink.examples.scala". I
am also fine with dropping the subpackages like "graph" or so. Not like we
have so many examples that we need subpackages ;-)

On Fri, Aug 14, 2015 at 11:01 AM, Till Rohrmann <[hidden email]>
wrote:

> In case of merging we would have to add a `scala` tag to some of the
> examples because some of the examples exist in both packages. The drawback
> of merging both examples modules is that we'll have another mixed
> Java/Scala project. This might be bothering for the eclipse folks, but we
> already have that. It might make sense to keep the examples separated if
> they use different dependencies. Other than that, I think both of your
> proposals are fine.
>
> Slightly off-topic: What we could also do is to move the shared components
> (e.g. the testing data) to a distinct module (flink-examples/example-data).
> But since the generated scala job jars only contain the java data classes
> this won't change much at all.
>
> On Fri, Aug 14, 2015 at 10:52 AM, Stephan Ewen <[hidden email]> wrote:
>
> > +1 for proposal (2)
> >
> > On Fri, Aug 14, 2015 at 10:28 AM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi Liang Chen,
> > >
> > > thanks for starting this discussion.
> > >
> > > I have no strong preference for one of your proposals. Both are fine
> with
> > > me.
> > >
> > > What would proposal 2 mean if we add examples for other API languages,
> > for
> > > example if the Python API moves out of staging?
> > > I guess it shouldn't be a problem, because all examples are
> > self-contained.
> > > Any other thoughts on this?
> > >
> > > Best, Fabian
> > >
> > > 2015-08-14 0:32 GMT+02:00 Liang Chen <[hidden email]>:
> > >
> > > > Summary:Simplify names of example program JARs
> > > > Key:FLINK-2493, URL:https://issues.apache.org/jira/browse/FLINK-2493
> > > > Issue Type: Improvement
> > > > Components:Examples
> > > >
> > > > Propose to simplify the name to "examples/ConnectedComponents.jar"
> > rather
> > > > than
> examples/flink-java-examples-0.10-SNAPSHOT-ConnectedComponents.jar
> > > >
> > > > Two proposals, wants to discuss:
> > > > Proposal 1:
> > > > Change the pom of "flink-java-examples", use executed-example name as
> > > > "finalName" to replace "default
> > > finalName(artifactId+version)+classifier".
> > > > keep two projects(flink-java-examples,flink-scala-examples) under
> > > > "flink-examples".
> > > >
> > > > Proposal 2:Combine "flink-java-examples" and "flink-scala-examples"
> > > project
> > > > to one project with "flink-examples", because the Scala example
> mostly
> > > need
> > > > import  Java example' util/*data*.class, work together.
> > > > this would be no any impaction for examples.
> > > >
> > > > please give your ideas, any input would be very appreciated.
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/jira-FLINK-2493-Simplify-names-of-example-program-JARs-tp7495.html
> > > > Sent from the Apache Flink Mailing List archive. mailing list archive
> > at
> > > > Nabble.com.
> > > >
> > >
> >
>

[Till Rohrmann]

Sounds good to me +1

On Fri, Aug 14, 2015 at 11:06 AM, Stephan Ewen <[hidden email]> wrote:

> Concerning mixed projects: Should be okay even for Eclipse users, because
> no Scala code is referenced from Java code (that is the tricky part).
> Even for someone without a Scala IDE plugin, they would simply not
> interpret the "src/main/scala" directory, which should make this appear
> like a pure Java project to them.
>
> Concerning name overlap: How about we put them in different packages:
> "org.apache.flink.examples.java" and "org.apache.flink.examples.scala". I
> am also fine with dropping the subpackages like "graph" or so. Not like we
> have so many examples that we need subpackages ;-)
>
> On Fri, Aug 14, 2015 at 11:01 AM, Till Rohrmann <[hidden email]>
> wrote:
>
> > In case of merging we would have to add a `scala` tag to some of the
> > examples because some of the examples exist in both packages. The
> drawback
> > of merging both examples modules is that we'll have another mixed
> > Java/Scala project. This might be bothering for the eclipse folks, but we
> > already have that. It might make sense to keep the examples separated if
> > they use different dependencies. Other than that, I think both of your
> > proposals are fine.
> >
> > Slightly off-topic: What we could also do is to move the shared
> components
> > (e.g. the testing data) to a distinct module
> (flink-examples/example-data).
> > But since the generated scala job jars only contain the java data classes
> > this won't change much at all.
> >
> > On Fri, Aug 14, 2015 at 10:52 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > > +1 for proposal (2)
> > >
> > > On Fri, Aug 14, 2015 at 10:28 AM, Fabian Hueske <[hidden email]>
> > wrote:
> > >
> > > > Hi Liang Chen,
> > > >
> > > > thanks for starting this discussion.
> > > >
> > > > I have no strong preference for one of your proposals. Both are fine
> > with
> > > > me.
> > > >
> > > > What would proposal 2 mean if we add examples for other API
> languages,
> > > for
> > > > example if the Python API moves out of staging?
> > > > I guess it shouldn't be a problem, because all examples are
> > > self-contained.
> > > > Any other thoughts on this?
> > > >
> > > > Best, Fabian
> > > >
> > > > 2015-08-14 0:32 GMT+02:00 Liang Chen <[hidden email]>:
> > > >
> > > > > Summary:Simplify names of example program JARs
> > > > > Key:FLINK-2493, URL:
> https://issues.apache.org/jira/browse/FLINK-2493
> > > > > Issue Type: Improvement
> > > > > Components:Examples
> > > > >
> > > > > Propose to simplify the name to "examples/ConnectedComponents.jar"
> > > rather
> > > > > than
> > examples/flink-java-examples-0.10-SNAPSHOT-ConnectedComponents.jar
> > > > >
> > > > > Two proposals, wants to discuss:
> > > > > Proposal 1:
> > > > > Change the pom of "flink-java-examples", use executed-example name
> as
> > > > > "finalName" to replace "default
> > > > finalName(artifactId+version)+classifier".
> > > > > keep two projects(flink-java-examples,flink-scala-examples) under
> > > > > "flink-examples".
> > > > >
> > > > > Proposal 2:Combine "flink-java-examples" and "flink-scala-examples"
> > > > project
> > > > > to one project with "flink-examples", because the Scala example
> > mostly
> > > > need
> > > > > import  Java example' util/*data*.class, work together.
> > > > > this would be no any impaction for examples.
> > > > >
> > > > > please give your ideas, any input would be very appreciated.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > View this message in context:
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/jira-FLINK-2493-Simplify-names-of-example-program-JARs-tp7495.html
> > > > > Sent from the Apache Flink Mailing List archive. mailing list
> archive
> > > at
> > > > > Nabble.com.
> > > > >
> > > >
> > >
> >
>

[Liang Chen]

Ya,If takes proposal 2, the name overlap is one issue from java and scala examples.
Your suggestion sounds good,gives different package for java and scala examples.

but drops the subpackages, i have some different points: there are many examples use the similar directory structure such as:storm-compatibility-examples,streaming-examples..., it would be better to keep consistent.

I found many places taking reference from the two artifactId(flink-java-examples and flink-scala-examples).
For avoiding some unpredictable issues for new starters,  so i prefer to "proposal 1", keep two projects.

BTW, my personal opinion: currently scala be acknowledged by more and more developers, in future there may have some different examples for scala.

Thanks for all guys' very good input.

[Stephan Ewen]

Hi!

Can you explain what you mean by "BTW, my personal opinion: currently scala
be acknowledged by more and more developers, in future there may have some
different examples for scala." ?

It would be fine to have different Scala and Java examples, there is no
need to have them in sync (although the very basic ones should be present
in both Java and Scala).

If we have both as JARs, this would work well, no?

Greetings,
Stephan


On Wed, Aug 19, 2015 at 9:47 AM, Liang Chen <[hidden email]> wrote:

> Ya,If takes proposal 2, the the name overlap is one issue from java and
> scala
> examples.
> Your suggestion sounds good,gives different package for java and scala
> examples.
>
> but drops the subpackages, i have some different points: there are many
> examples use the similar directory structure such
> as:storm-compatibility-examples,streaming-examples..., it would be better
> to
> keep consistent.
>
> I found many places taking reference from the two
> artifactId(flink-java-examples and flink-scala-examples).
> For avoiding some unpredictable issues for new starters,  so i prefer to
> "proposal 1", keep two projects.
>
> BTW, my personal opinion: currently scala be acknowledged by more and more
> developers, in future there may have some different examples for scala.
>
> Thanks for all guys' very good input.
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/jira-FLINK-2493-Simplify-names-of-example-program-JARs-tp7495p7571.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

[Liang Chen]

Sorry,made you confusion.
I meant, may only use scala to write some examples(for example:machine learning) in future. (java doesn't include these examples).
-------------------------------------------------------------------------------------------
Can you explain what you mean by "BTW, my personal opinion: currently scala
be acknowledged by more and more developers, in future there may have some
different examples for scala." ?

[Liang Chen]

Thanks for everyone's input.
For keeping java and scala examples separately, will take proposal 1 to change POM.
meanwhile,will make "flink-streaming-examples and flink-storm-compatibility-example" changes too according to mbalassi's suggestion.

One question:  As per pom.xml of scala/storm/streaming(the above mentioned examples) description,   only unpacking the "flink-java-examples-0.10-SNAPSHOT.jar"  for reference java examples' *data.class, doesn't unpack other *.jar file.  So through setting finalName to replace classifier for changing  the examples' name(for example: from "flink-java-examples-0.10-SNAPSHOT-KMeans.jar" to "KMeans.jar") , should not impact other program to unpack the "flink-java-examples-0.10-SNAPSHOT.jar", because there are no any change for the "flink-java-examples-0.10-SNAPSHOT.jar".

---------flink-scala-example reference java example error msg--------
[ERROR] /Users/chenliang/flink/flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala:148: error: not found: value KMeansData
[ERROR]       val points = KMeansData.POINTS map {
[ERROR]                    ^
[WARNING] /Users/chenliang/flink/flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala:161: warning: Type org.apache.flink.examples.scala.clustering.KMeans.Centroid is no POJO, has immutable fields: value y, value x.
[WARNING]         .map { x => new Centroid(x._1, x._2, x._3)}
[WARNING]              ^
[ERROR] /Users/chenliang/flink/flink-examples/flink-scala-examples/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala:164: error: not found: value KMeansData
[ERROR]       val centroids = KMeansData.CENTROIDS map {

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.9</version>
    <executions>
     <execution>
      <id>unpack</id>
      <phase>prepare-package</phase>
      <goals>
       <goal>unpack</goal>
      </goals>
      <configuration>
       <artifactItems>
        <artifactItem>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-java-examples</artifactId>
         <version>${project.version}</version>
         <type>jar</type>
         <overWrite>false</overWrite>
         <outputDirectory>${project.build.directory}/classes</outputDirectory>
         <includes>**/util/*Data*.class</includes>