Replacing JobManager with Scala implementation

[Daniel Warneke]

Stephan,

I agree that the questions “akka” and “Scala” should be treated
separately. Unfortunately, this is not how the discussion has been led
so far. Instead, the new akka RPC service is used to motivate the
necessity for Scala in the runtime core. I still don’t see that
necessity. I tried to find the implementation of the new akka RPC
service on github. The only code I found was from Asterios, but it looks
like he was perfectly able to encapsulate the whole akka RPC thing in 5
Java classes [1].

The second argument (Scala will attract new developers to the project)
is nothing but speculation. This might as well totally backfire and lead
to the opposite.

The only explanation I have for this push towards akka and Scala is that
there are already plans to expand the usage of akka way beyond pure RPC.
In this case, I feel these plans should be clearly articulated on the
dev list. A simple RPC service does not justify the proposed changes in
my opinion.

Best regards,

     Daniel

[1]
https://github.com/asteriosk/stratosphere/tree/akka_rpc/stratosphere-runtime/src/main/java/eu/stratosphere/nephele/ipc 




Am 04.09.2014 12:00, schrieb Stephan Ewen:

> Again, Java vs. Scala is a different question than akka or no akka.
>
> Java vs Scala is in the end a question of ease of use, and of adding a
> stronger Scala side to the project (for our own sake or for attracting
> interested developers).
>
> Akka is planned not simply as an rpc replacement alone, but for remote
> calls, asynchronous callbacks, failure detection, master failover, and a
> simpler concurrency model in the execution graph.
>
> I Actually think that the current rpc is limiting us, for example because
> of the lack of callback support, which makes the polling parts necessary.
>
> Stephan
>

[Ufuk Celebi-2]

Hey Daniel,

On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <[hidden email]> wrote:

> I agree that the questions “akka” and “Scala” should be treated
> separately. Unfortunately, this is not how the discussion has been led so
> far. Instead, the new akka RPC service is used to motivate the necessity
> for Scala in the runtime core. I still don’t see that necessity. I tried to
> find the implementation of the new akka RPC service on github. The only
> code I found was from Asterios, but it looks like he was perfectly able to
> encapsulate the whole akka RPC thing in 5 Java classes [1].
>

I don't think that anybody is talking about the necessity of Scala. Yes,
Akka and an actor based refactoring of core runtime parts result in a hard
dependency to Scala for the core (because Akka is written in Scala), but it
does *not* necessitate to do the refactoring itself in Scala, because there
is an Akka Java API as well.

Are you concerned with the dependency to Scala or with using Akka's Scala
API?

I think that Till started this thread and the [VOTE] exactly because he is
well aware that it is *not* necessary to do it in Scala. He sees good
reasons to do it in Scala and asks the community to vote on it. Again,
because he is aware that this is not a small or light weight change.

The second argument (Scala will attract new developers to the project) is
> nothing but speculation. This might as well totally backfire and lead to
> the opposite.
>

I agree that this point is speculative and both outlined outcomes (attract
or repel developers) are possible. But it is also not the only argument
that has been raised in favor of Scala. Other more technical (not
speculative) points have been given. It is the goal of the vote to find a
consensus about whether these points are sufficient or not.


> The only explanation I have for this push towards akka and Scala is that
> there are already plans to expand the usage of akka way beyond pure RPC. In
> this case, I feel these plans should be clearly articulated on the dev
> list. A simple RPC service does not justify the proposed changes in my
> opinion.
>

There is no push for Scala. It's a vote. And the reasons for going for Akka
have been repeated a few times by now.

I think the way that Asterios initially introduced Akka beneath the
existing RPC proxy service (independently of whether he did in Scala or
Java) would not allow us to make use of central features of Akka (some of
which Till and Stephan outlined).

Best wishes,

Ufuk

[Kostas Tzoumas-2]

Hi Daniel,

+1 on the argument about attracting developers being irrelevant, the
argument can work both ways and is very brittle

The reasons for using Akka as a library (irrespective of the programming
language) have been clearly articulated in my opinion by Stephan and Till
in this thread.

The reasons for using the Scala Akka API versus the Java Akka API is simply
ease and speed of development, as Scala is a better language for this task.
Perhaps Till could expand the argumentation a bit more here, but I suspect
that a language with pattern matching is a good fit for message
passing-like systems.

Kostas



On Fri, Sep 5, 2014 at 12:38 AM, Ufuk Celebi <[hidden email]> wrote:

> Hey Daniel,
>
> On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <[hidden email]>
> wrote:
>
> > I agree that the questions “akka” and “Scala” should be treated
> > separately. Unfortunately, this is not how the discussion has been led so
> > far. Instead, the new akka RPC service is used to motivate the necessity
> > for Scala in the runtime core. I still don’t see that necessity. I tried
> to
> > find the implementation of the new akka RPC service on github. The only
> > code I found was from Asterios, but it looks like he was perfectly able
> to
> > encapsulate the whole akka RPC thing in 5 Java classes [1].
> >
>
> I don't think that anybody is talking about the necessity of Scala. Yes,
> Akka and an actor based refactoring of core runtime parts result in a hard
> dependency to Scala for the core (because Akka is written in Scala), but it
> does *not* necessitate to do the refactoring itself in Scala, because there
> is an Akka Java API as well.
>
> Are you concerned with the dependency to Scala or with using Akka's Scala
> API?
>
> I think that Till started this thread and the [VOTE] exactly because he is
> well aware that it is *not* necessary to do it in Scala. He sees good
> reasons to do it in Scala and asks the community to vote on it. Again,
> because he is aware that this is not a small or light weight change.
>
> The second argument (Scala will attract new developers to the project) is
> > nothing but speculation. This might as well totally backfire and lead to
> > the opposite.
> >
>
> I agree that this point is speculative and both outlined outcomes (attract
> or repel developers) are possible. But it is also not the only argument
> that has been raised in favor of Scala. Other more technical (not
> speculative) points have been given. It is the goal of the vote to find a
> consensus about whether these points are sufficient or not.
>
>
> > The only explanation I have for this push towards akka and Scala is that
> > there are already plans to expand the usage of akka way beyond pure RPC.
> In
> > this case, I feel these plans should be clearly articulated on the dev
> > list. A simple RPC service does not justify the proposed changes in my
> > opinion.
> >
>
> There is no push for Scala. It's a vote. And the reasons for going for Akka
> have been repeated a few times by now.
>
> I think the way that Asterios initially introduced Akka beneath the
> existing RPC proxy service (independently of whether he did in Scala or
> Java) would not allow us to make use of central features of Akka (some of
> which Till and Stephan outlined).
>
> Best wishes,
>
> Ufuk
>

[Till Rohrmann]

Hi Daniel,

the genesis of my proposal was that I resumed Asterios' effort to introduce
Akka to handle our RPC calls. Asterios idea was to incorporate Akka into
Flink transparently. That meant to use Akka for the MethodInvoker of the
proxies and as the server to handle the calls. While rebasing it on the
latest master of Flink I noticed that we left out some potential of Akka by
pursuing this approach. Among others, this includes the supervision,
monitoring and exception forwarding mechanism. By making the JobManager and
TaskManager (and possibly other components if it proves worth) actors, we
would get this for free, without having to go through the proxies, method
invoker and call actors. Furthermore, I think that this adds unnecessary
complexity. If we use Akka, why not properly? At this point you are right
that we did not vote on using Akka and I did not think about it. I thought
that by continuing Asterios work, it would comply with the project.

However, making JobManagers and TaskManagers actors comes at the price of
rewriting them, which is unavoidable. Since Akka offers Scala as well as
Java bindings, it would be possible to do it with both. Since Scala
supports pattern matching, lambdas and embraces more the functional
paradigm, it is in my view better suited for implementing an actor based
RPC system. With Java, the code will become more verbose and probably not
so neat to read.

For example, for each message it would be necessary to create a Java class
with the members, getters and setters whereas in Scala it is most of the
time a one line case class definition. For each call back it would be
necessary to implement an interface and overwrite the corresponding method
whereas for Scala it is just a lambda expression (to be precise an
anonymous function). For short call back functions we would have more Java
boiler plate code than actual function code.

In order to distinguish the different messages in Java, one would have a
long list of if(msg instanceof RegisterMessage){} else if (msg instanceof
SubmitTask){}.... and cannot directly access the members of the message
classes. In Scala it would be simply a pattern matching. Since both
languages are Turing-complete and even run in the same JVM, the differences
are thus conciseness and expressiveness. But in the end everything can also
be implemented in Java.

There is no functional argument for Scala just the advantages of a shorter
and probably more readable implementation. Since I will probably implement
it, I'd opt for the shorter solution and thus I started this vote. But
there is no necessity to use Scala here. If we decide for Akka, then Akka
will simply motivate the use of Scala but not enforce it.

Best regards,

Till


On Fri, Sep 5, 2014 at 11:24 AM, Kostas Tzoumas <[hidden email]> wrote:

> Hi Daniel,
>
> +1 on the argument about attracting developers being irrelevant, the
> argument can work both ways and is very brittle
>
> The reasons for using Akka as a library (irrespective of the programming
> language) have been clearly articulated in my opinion by Stephan and Till
> in this thread.
>
> The reasons for using the Scala Akka API versus the Java Akka API is simply
> ease and speed of development, as Scala is a better language for this task.
> Perhaps Till could expand the argumentation a bit more here, but I suspect
> that a language with pattern matching is a good fit for message
> passing-like systems.
>
> Kostas
>
>
>
> On Fri, Sep 5, 2014 at 12:38 AM, Ufuk Celebi <[hidden email]> wrote:
>
> > Hey Daniel,
> >
> > On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <[hidden email]>
> > wrote:
> >
> > > I agree that the questions “akka” and “Scala” should be treated
> > > separately. Unfortunately, this is not how the discussion has been led
> so
> > > far. Instead, the new akka RPC service is used to motivate the
> necessity
> > > for Scala in the runtime core. I still don’t see that necessity. I
> tried
> > to
> > > find the implementation of the new akka RPC service on github. The only
> > > code I found was from Asterios, but it looks like he was perfectly able
> > to
> > > encapsulate the whole akka RPC thing in 5 Java classes [1].
> > >
> >
> > I don't think that anybody is talking about the necessity of Scala. Yes,
> > Akka and an actor based refactoring of core runtime parts result in a
> hard
> > dependency to Scala for the core (because Akka is written in Scala), but
> it
> > does *not* necessitate to do the refactoring itself in Scala, because
> there
> > is an Akka Java API as well.
> >
> > Are you concerned with the dependency to Scala or with using Akka's Scala
> > API?
> >
> > I think that Till started this thread and the [VOTE] exactly because he
> is
> > well aware that it is *not* necessary to do it in Scala. He sees good
> > reasons to do it in Scala and asks the community to vote on it. Again,
> > because he is aware that this is not a small or light weight change.
> >
> > The second argument (Scala will attract new developers to the project) is
> > > nothing but speculation. This might as well totally backfire and lead
> to
> > > the opposite.
> > >
> >
> > I agree that this point is speculative and both outlined outcomes
> (attract
> > or repel developers) are possible. But it is also not the only argument
> > that has been raised in favor of Scala. Other more technical (not
> > speculative) points have been given. It is the goal of the vote to find a
> > consensus about whether these points are sufficient or not.
> >
> >
> > > The only explanation I have for this push towards akka and Scala is
> that
> > > there are already plans to expand the usage of akka way beyond pure
> RPC.
> > In
> > > this case, I feel these plans should be clearly articulated on the dev
> > > list. A simple RPC service does not justify the proposed changes in my
> > > opinion.
> > >
> >
> > There is no push for Scala. It's a vote. And the reasons for going for
> Akka
> > have been repeated a few times by now.
> >
> > I think the way that Asterios initially introduced Akka beneath the
> > existing RPC proxy service (independently of whether he did in Scala or
> > Java) would not allow us to make use of central features of Akka (some of
> > which Till and Stephan outlined).
> >
> > Best wishes,
> >
> > Ufuk
> >
>

[Stephan Ewen]

Okay, to make things clearer, I will start a separate thread for the
proposed use of Akka, which talks only about the Akka/RPC question, because
things keep getting confused.

Let's postpone this thread (Java vs Scala) until we resolved the Akka
question first.


On Fri, Sep 5, 2014 at 12:10 PM, Till Rohrmann <[hidden email]> wrote:

> Hi Daniel,
>
> the genesis of my proposal was that I resumed Asterios' effort to introduce
> Akka to handle our RPC calls. Asterios idea was to incorporate Akka into
> Flink transparently. That meant to use Akka for the MethodInvoker of the
> proxies and as the server to handle the calls. While rebasing it on the
> latest master of Flink I noticed that we left out some potential of Akka by
> pursuing this approach. Among others, this includes the supervision,
> monitoring and exception forwarding mechanism. By making the JobManager and
> TaskManager (and possibly other components if it proves worth) actors, we
> would get this for free, without having to go through the proxies, method
> invoker and call actors. Furthermore, I think that this adds unnecessary
> complexity. If we use Akka, why not properly? At this point you are right
> that we did not vote on using Akka and I did not think about it. I thought
> that by continuing Asterios work, it would comply with the project.
>
> However, making JobManagers and TaskManagers actors comes at the price of
> rewriting them, which is unavoidable. Since Akka offers Scala as well as
> Java bindings, it would be possible to do it with both. Since Scala
> supports pattern matching, lambdas and embraces more the functional
> paradigm, it is in my view better suited for implementing an actor based
> RPC system. With Java, the code will become more verbose and probably not
> so neat to read.
>
> For example, for each message it would be necessary to create a Java class
> with the members, getters and setters whereas in Scala it is most of the
> time a one line case class definition. For each call back it would be
> necessary to implement an interface and overwrite the corresponding method
> whereas for Scala it is just a lambda expression (to be precise an
> anonymous function). For short call back functions we would have more Java
> boiler plate code than actual function code.
>
> In order to distinguish the different messages in Java, one would have a
> long list of if(msg instanceof RegisterMessage){} else if (msg instanceof
> SubmitTask){}.... and cannot directly access the members of the message
> classes. In Scala it would be simply a pattern matching. Since both
> languages are Turing-complete and even run in the same JVM, the differences
> are thus conciseness and expressiveness. But in the end everything can also
> be implemented in Java.
>
> There is no functional argument for Scala just the advantages of a shorter
> and probably more readable implementation. Since I will probably implement
> it, I'd opt for the shorter solution and thus I started this vote. But
> there is no necessity to use Scala here. If we decide for Akka, then Akka
> will simply motivate the use of Scala but not enforce it.
>
> Best regards,
>
> Till
>
>
> On Fri, Sep 5, 2014 at 11:24 AM, Kostas Tzoumas <[hidden email]>
> wrote:
>
> > Hi Daniel,
> >
> > +1 on the argument about attracting developers being irrelevant, the
> > argument can work both ways and is very brittle
> >
> > The reasons for using Akka as a library (irrespective of the programming
> > language) have been clearly articulated in my opinion by Stephan and Till
> > in this thread.
> >
> > The reasons for using the Scala Akka API versus the Java Akka API is
> simply
> > ease and speed of development, as Scala is a better language for this
> task.
> > Perhaps Till could expand the argumentation a bit more here, but I
> suspect
> > that a language with pattern matching is a good fit for message
> > passing-like systems.
> >
> > Kostas
> >
> >
> >
> > On Fri, Sep 5, 2014 at 12:38 AM, Ufuk Celebi <[hidden email]> wrote:
> >
> > > Hey Daniel,
> > >
> > > On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <[hidden email]>
> > > wrote:
> > >
> > > > I agree that the questions “akka” and “Scala” should be treated
> > > > separately. Unfortunately, this is not how the discussion has been
> led
> > so
> > > > far. Instead, the new akka RPC service is used to motivate the
> > necessity
> > > > for Scala in the runtime core. I still don’t see that necessity. I
> > tried
> > > to
> > > > find the implementation of the new akka RPC service on github. The
> only
> > > > code I found was from Asterios, but it looks like he was perfectly
> able
> > > to
> > > > encapsulate the whole akka RPC thing in 5 Java classes [1].
> > > >
> > >
> > > I don't think that anybody is talking about the necessity of Scala.
> Yes,
> > > Akka and an actor based refactoring of core runtime parts result in a
> > hard
> > > dependency to Scala for the core (because Akka is written in Scala),
> but
> > it
> > > does *not* necessitate to do the refactoring itself in Scala, because
> > there
> > > is an Akka Java API as well.
> > >
> > > Are you concerned with the dependency to Scala or with using Akka's
> Scala
> > > API?
> > >
> > > I think that Till started this thread and the [VOTE] exactly because he
> > is
> > > well aware that it is *not* necessary to do it in Scala. He sees good
> > > reasons to do it in Scala and asks the community to vote on it. Again,
> > > because he is aware that this is not a small or light weight change.
> > >
> > > The second argument (Scala will attract new developers to the project)
> is
> > > > nothing but speculation. This might as well totally backfire and lead
> > to
> > > > the opposite.
> > > >
> > >
> > > I agree that this point is speculative and both outlined outcomes
> > (attract
> > > or repel developers) are possible. But it is also not the only argument
> > > that has been raised in favor of Scala. Other more technical (not
> > > speculative) points have been given. It is the goal of the vote to
> find a
> > > consensus about whether these points are sufficient or not.
> > >
> > >
> > > > The only explanation I have for this push towards akka and Scala is
> > that
> > > > there are already plans to expand the usage of akka way beyond pure
> > RPC.
> > > In
> > > > this case, I feel these plans should be clearly articulated on the
> dev
> > > > list. A simple RPC service does not justify the proposed changes in
> my
> > > > opinion.
> > > >
> > >
> > > There is no push for Scala. It's a vote. And the reasons for going for
> > Akka
> > > have been repeated a few times by now.
> > >
> > > I think the way that Asterios initially introduced Akka beneath the
> > > existing RPC proxy service (independently of whether he did in Scala or
> > > Java) would not allow us to make use of central features of Akka (some
> of
> > > which Till and Stephan outlined).
> > >
> > > Best wishes,
> > >
> > > Ufuk
> > >
> >
>

[Ufuk Celebi-2]

OK. Thanks for starting the new thread.

I would say that we then [CANCEL] the ongoing vote about Scala vs. Java
until the Akka issue is resolved.


On Fri, Sep 5, 2014 at 12:54 PM, Stephan Ewen <[hidden email]> wrote:

> Okay, to make things clearer, I will start a separate thread for the
> proposed use of Akka, which talks only about the Akka/RPC question, because
> things keep getting confused.
>
> Let's postpone this thread (Java vs Scala) until we resolved the Akka
> question first.
>
>
> On Fri, Sep 5, 2014 at 12:10 PM, Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi Daniel,
> >
> > the genesis of my proposal was that I resumed Asterios' effort to
> introduce
> > Akka to handle our RPC calls. Asterios idea was to incorporate Akka into
> > Flink transparently. That meant to use Akka for the MethodInvoker of the
> > proxies and as the server to handle the calls. While rebasing it on the
> > latest master of Flink I noticed that we left out some potential of Akka
> by
> > pursuing this approach. Among others, this includes the supervision,
> > monitoring and exception forwarding mechanism. By making the JobManager
> and
> > TaskManager (and possibly other components if it proves worth) actors, we
> > would get this for free, without having to go through the proxies, method
> > invoker and call actors. Furthermore, I think that this adds unnecessary
> > complexity. If we use Akka, why not properly? At this point you are right
> > that we did not vote on using Akka and I did not think about it. I
> thought
> > that by continuing Asterios work, it would comply with the project.
> >
> > However, making JobManagers and TaskManagers actors comes at the price of
> > rewriting them, which is unavoidable. Since Akka offers Scala as well as
> > Java bindings, it would be possible to do it with both. Since Scala
> > supports pattern matching, lambdas and embraces more the functional
> > paradigm, it is in my view better suited for implementing an actor based
> > RPC system. With Java, the code will become more verbose and probably not
> > so neat to read.
> >
> > For example, for each message it would be necessary to create a Java
> class
> > with the members, getters and setters whereas in Scala it is most of the
> > time a one line case class definition. For each call back it would be
> > necessary to implement an interface and overwrite the corresponding
> method
> > whereas for Scala it is just a lambda expression (to be precise an
> > anonymous function). For short call back functions we would have more
> Java
> > boiler plate code than actual function code.
> >
> > In order to distinguish the different messages in Java, one would have a
> > long list of if(msg instanceof RegisterMessage){} else if (msg instanceof
> > SubmitTask){}.... and cannot directly access the members of the message
> > classes. In Scala it would be simply a pattern matching. Since both
> > languages are Turing-complete and even run in the same JVM, the
> differences
> > are thus conciseness and expressiveness. But in the end everything can
> also
> > be implemented in Java.
> >
> > There is no functional argument for Scala just the advantages of a
> shorter
> > and probably more readable implementation. Since I will probably
> implement
> > it, I'd opt for the shorter solution and thus I started this vote. But
> > there is no necessity to use Scala here. If we decide for Akka, then Akka
> > will simply motivate the use of Scala but not enforce it.
> >
> > Best regards,
> >
> > Till
> >
> >
> > On Fri, Sep 5, 2014 at 11:24 AM, Kostas Tzoumas <[hidden email]>
> > wrote:
> >
> > > Hi Daniel,
> > >
> > > +1 on the argument about attracting developers being irrelevant, the
> > > argument can work both ways and is very brittle
> > >
> > > The reasons for using Akka as a library (irrespective of the
> programming
> > > language) have been clearly articulated in my opinion by Stephan and
> Till
> > > in this thread.
> > >
> > > The reasons for using the Scala Akka API versus the Java Akka API is
> > simply
> > > ease and speed of development, as Scala is a better language for this
> > task.
> > > Perhaps Till could expand the argumentation a bit more here, but I
> > suspect
> > > that a language with pattern matching is a good fit for message
> > > passing-like systems.
> > >
> > > Kostas
> > >
> > >
> > >
> > > On Fri, Sep 5, 2014 at 12:38 AM, Ufuk Celebi <[hidden email]> wrote:
> > >
> > > > Hey Daniel,
> > > >
> > > > On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <[hidden email]>
> > > > wrote:
> > > >
> > > > > I agree that the questions “akka” and “Scala” should be treated
> > > > > separately. Unfortunately, this is not how the discussion has been
> > led
> > > so
> > > > > far. Instead, the new akka RPC service is used to motivate the
> > > necessity
> > > > > for Scala in the runtime core. I still don’t see that necessity. I
> > > tried
> > > > to
> > > > > find the implementation of the new akka RPC service on github. The
> > only
> > > > > code I found was from Asterios, but it looks like he was perfectly
> > able
> > > > to
> > > > > encapsulate the whole akka RPC thing in 5 Java classes [1].
> > > > >
> > > >
> > > > I don't think that anybody is talking about the necessity of Scala.
> > Yes,
> > > > Akka and an actor based refactoring of core runtime parts result in a
> > > hard
> > > > dependency to Scala for the core (because Akka is written in Scala),
> > but
> > > it
> > > > does *not* necessitate to do the refactoring itself in Scala, because
> > > there
> > > > is an Akka Java API as well.
> > > >
> > > > Are you concerned with the dependency to Scala or with using Akka's
> > Scala
> > > > API?
> > > >
> > > > I think that Till started this thread and the [VOTE] exactly because
> he
> > > is
> > > > well aware that it is *not* necessary to do it in Scala. He sees good
> > > > reasons to do it in Scala and asks the community to vote on it.
> Again,
> > > > because he is aware that this is not a small or light weight change.
> > > >
> > > > The second argument (Scala will attract new developers to the
> project)
> > is
> > > > > nothing but speculation. This might as well totally backfire and
> lead
> > > to
> > > > > the opposite.
> > > > >
> > > >
> > > > I agree that this point is speculative and both outlined outcomes
> > > (attract
> > > > or repel developers) are possible. But it is also not the only
> argument
> > > > that has been raised in favor of Scala. Other more technical (not
> > > > speculative) points have been given. It is the goal of the vote to
> > find a
> > > > consensus about whether these points are sufficient or not.
> > > >
> > > >
> > > > > The only explanation I have for this push towards akka and Scala is
> > > that
> > > > > there are already plans to expand the usage of akka way beyond pure
> > > RPC.
> > > > In
> > > > > this case, I feel these plans should be clearly articulated on the
> > dev
> > > > > list. A simple RPC service does not justify the proposed changes in
> > my
> > > > > opinion.
> > > > >
> > > >
> > > > There is no push for Scala. It's a vote. And the reasons for going
> for
> > > Akka
> > > > have been repeated a few times by now.
> > > >
> > > > I think the way that Asterios initially introduced Akka beneath the
> > > > existing RPC proxy service (independently of whether he did in Scala
> or
> > > > Java) would not allow us to make use of central features of Akka
> (some
> > of
> > > > which Till and Stephan outlined).
> > > >
> > > > Best wishes,
> > > >
> > > > Ufuk
> > > >
> > >
> >
>

[till.rohrmann]

I'll cancel the vote.


On Fri, Sep 5, 2014 at 1:43 PM, Ufuk Celebi <[hidden email]> wrote:

> OK. Thanks for starting the new thread.
>
> I would say that we then [CANCEL] the ongoing vote about Scala vs. Java
> until the Akka issue is resolved.
>
>
> On Fri, Sep 5, 2014 at 12:54 PM, Stephan Ewen <[hidden email]> wrote:
>
> > Okay, to make things clearer, I will start a separate thread for the
> > proposed use of Akka, which talks only about the Akka/RPC question,
> because
> > things keep getting confused.
> >
> > Let's postpone this thread (Java vs Scala) until we resolved the Akka
> > question first.
> >
> >
> > On Fri, Sep 5, 2014 at 12:10 PM, Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Hi Daniel,
> > >
> > > the genesis of my proposal was that I resumed Asterios' effort to
> > introduce
> > > Akka to handle our RPC calls. Asterios idea was to incorporate Akka
> into
> > > Flink transparently. That meant to use Akka for the MethodInvoker of
> the
> > > proxies and as the server to handle the calls. While rebasing it on the
> > > latest master of Flink I noticed that we left out some potential of
> Akka
> > by
> > > pursuing this approach. Among others, this includes the supervision,
> > > monitoring and exception forwarding mechanism. By making the JobManager
> > and
> > > TaskManager (and possibly other components if it proves worth) actors,
> we
> > > would get this for free, without having to go through the proxies,
> method
> > > invoker and call actors. Furthermore, I think that this adds
> unnecessary
> > > complexity. If we use Akka, why not properly? At this point you are
> right
> > > that we did not vote on using Akka and I did not think about it. I
> > thought
> > > that by continuing Asterios work, it would comply with the project.
> > >
> > > However, making JobManagers and TaskManagers actors comes at the price
> of
> > > rewriting them, which is unavoidable. Since Akka offers Scala as well
> as
> > > Java bindings, it would be possible to do it with both. Since Scala
> > > supports pattern matching, lambdas and embraces more the functional
> > > paradigm, it is in my view better suited for implementing an actor
> based
> > > RPC system. With Java, the code will become more verbose and probably
> not
> > > so neat to read.
> > >
> > > For example, for each message it would be necessary to create a Java
> > class
> > > with the members, getters and setters whereas in Scala it is most of
> the
> > > time a one line case class definition. For each call back it would be
> > > necessary to implement an interface and overwrite the corresponding
> > method
> > > whereas for Scala it is just a lambda expression (to be precise an
> > > anonymous function). For short call back functions we would have more
> > Java
> > > boiler plate code than actual function code.
> > >
> > > In order to distinguish the different messages in Java, one would have
> a
> > > long list of if(msg instanceof RegisterMessage){} else if (msg
> instanceof
> > > SubmitTask){}.... and cannot directly access the members of the message
> > > classes. In Scala it would be simply a pattern matching. Since both
> > > languages are Turing-complete and even run in the same JVM, the
> > differences
> > > are thus conciseness and expressiveness. But in the end everything can
> > also
> > > be implemented in Java.
> > >
> > > There is no functional argument for Scala just the advantages of a
> > shorter
> > > and probably more readable implementation. Since I will probably
> > implement
> > > it, I'd opt for the shorter solution and thus I started this vote. But
> > > there is no necessity to use Scala here. If we decide for Akka, then
> Akka
> > > will simply motivate the use of Scala but not enforce it.
> > >
> > > Best regards,
> > >
> > > Till
> > >
> > >
> > > On Fri, Sep 5, 2014 at 11:24 AM, Kostas Tzoumas <[hidden email]>
> > > wrote:
> > >
> > > > Hi Daniel,
> > > >
> > > > +1 on the argument about attracting developers being irrelevant, the
> > > > argument can work both ways and is very brittle
> > > >
> > > > The reasons for using Akka as a library (irrespective of the
> > programming
> > > > language) have been clearly articulated in my opinion by Stephan and
> > Till
> > > > in this thread.
> > > >
> > > > The reasons for using the Scala Akka API versus the Java Akka API is
> > > simply
> > > > ease and speed of development, as Scala is a better language for this
> > > task.
> > > > Perhaps Till could expand the argumentation a bit more here, but I
> > > suspect
> > > > that a language with pattern matching is a good fit for message
> > > > passing-like systems.
> > > >
> > > > Kostas
> > > >
> > > >
> > > >
> > > > On Fri, Sep 5, 2014 at 12:38 AM, Ufuk Celebi <[hidden email]> wrote:
> > > >
> > > > > Hey Daniel,
> > > > >
> > > > > On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > I agree that the questions “akka” and “Scala” should be treated
> > > > > > separately. Unfortunately, this is not how the discussion has
> been
> > > led
> > > > so
> > > > > > far. Instead, the new akka RPC service is used to motivate the
> > > > necessity
> > > > > > for Scala in the runtime core. I still don’t see that necessity.
> I
> > > > tried
> > > > > to
> > > > > > find the implementation of the new akka RPC service on github.
> The
> > > only
> > > > > > code I found was from Asterios, but it looks like he was
> perfectly
> > > able
> > > > > to
> > > > > > encapsulate the whole akka RPC thing in 5 Java classes [1].
> > > > > >
> > > > >
> > > > > I don't think that anybody is talking about the necessity of Scala.
> > > Yes,
> > > > > Akka and an actor based refactoring of core runtime parts result
> in a
> > > > hard
> > > > > dependency to Scala for the core (because Akka is written in
> Scala),
> > > but
> > > > it
> > > > > does *not* necessitate to do the refactoring itself in Scala,
> because
> > > > there
> > > > > is an Akka Java API as well.
> > > > >
> > > > > Are you concerned with the dependency to Scala or with using Akka's
> > > Scala
> > > > > API?
> > > > >
> > > > > I think that Till started this thread and the [VOTE] exactly
> because
> > he
> > > > is
> > > > > well aware that it is *not* necessary to do it in Scala. He sees
> good
> > > > > reasons to do it in Scala and asks the community to vote on it.
> > Again,
> > > > > because he is aware that this is not a small or light weight
> change.
> > > > >
> > > > > The second argument (Scala will attract new developers to the
> > project)
> > > is
> > > > > > nothing but speculation. This might as well totally backfire and
> > lead
> > > > to
> > > > > > the opposite.
> > > > > >
> > > > >
> > > > > I agree that this point is speculative and both outlined outcomes
> > > > (attract
> > > > > or repel developers) are possible. But it is also not the only
> > argument
> > > > > that has been raised in favor of Scala. Other more technical (not
> > > > > speculative) points have been given. It is the goal of the vote to
> > > find a
> > > > > consensus about whether these points are sufficient or not.
> > > > >
> > > > >
> > > > > > The only explanation I have for this push towards akka and Scala
> is
> > > > that
> > > > > > there are already plans to expand the usage of akka way beyond
> pure
> > > > RPC.
> > > > > In
> > > > > > this case, I feel these plans should be clearly articulated on
> the
> > > dev
> > > > > > list. A simple RPC service does not justify the proposed changes
> in
> > > my
> > > > > > opinion.
> > > > > >
> > > > >
> > > > > There is no push for Scala. It's a vote. And the reasons for going
> > for
> > > > Akka
> > > > > have been repeated a few times by now.
> > > > >
> > > > > I think the way that Asterios initially introduced Akka beneath the
> > > > > existing RPC proxy service (independently of whether he did in
> Scala
> > or
> > > > > Java) would not allow us to make use of central features of Akka
> > (some
> > > of
> > > > > which Till and Stephan outlined).
> > > > >
> > > > > Best wishes,
> > > > >
> > > > > Ufuk
> > > > >
> > > >
> > >
> >
>