(DEPRECATED) Apache Flink Mailing List archive.

Apache Flink:ProgramInvocationException on Yarn

Classic

List

Threaded

30 messages Options

Alexey Sapozhnikov

Re: Apache Flink:ProgramInvocationException on Yarn

Here is the exception from the moment we tried to put in
jobmanager.rpc.address the hostname of the machine which is ip-172-36-98
looks like it doesnt recognize this address.
Why it doesnt support "0.0.0.0"????

13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
-
--------------------------------------------------------------------------------
13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
- Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf
13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
- Security is not enabled. Starting non-authenticated JobManager.
13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
- Starting JobManager
13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
- Starting JobManager actor system at ip-172-36-98:6123.
13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
- Failed to run JobManager.
java.lang.Exception: Could not create JobManager actor system
at
org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
at
org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
at
org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
@
21,1
71%

On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <[hidden email]> wrote:

> How is Flink crashing when you start it on the Linux machine in Amazon?
>
> Can you post the exception here?
>
> On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <[hidden email]>
> wrote:
>
> > Hello Stephan.
> >
> > We run this Linux machine on Amazon, which I predict, most of the people
> > will do.
> > We tried to put "0.0.0.0" or Public IP of the machine- Flink crashes on
> > start, it doesnt recognize himself.
> > It is very strange that it doesnt work with 0.0.0.0- basically this is a
> > way in Java to make it being seen widely.
> > We tried to put there the hostname - what you get from "hostname"
> command.
> > It crashes.
> > It works only with "localhost" and works only locally
> > So what you suggest we will put there so the remote client could connect.
> > ?
> >
> > Best regards
> >
> > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Do you start Flink via YARN? In that case the "jobmanager.rpc.address"
> is
> > > not used, because YARN assigns containers/nodes.
> > >
> > > If you start Flink in "standalone" mode, this should be the address of
> > the
> > > node that runs the JobManager. It will be used as the host/IP that
> Flink
> > > binds to. The same host should then be used in the RemoteEnvironment.
> > >
> > > Stephan
> > >
> > >
> > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <[hidden email]>
> wrote:
> > >
> > > > Hello all.
> > > >
> > > > Firstly- thank you for your valuable advices.
> > > > We did some very fine tuned pinpoint test and comes to following
> > > > conclusions
> > > >
> > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > 2.Once we copy our Java client program directy to the machine and run
> > it
> > > > directly there it worked very good
> > > > The program is
> > > >
> > > > .....
> > > >
> > > > ExecutionEnvironment envRemote
> > > > =ExecutionEnvironment.createRemoteEnvironment("localhost", 6123,
> > > > "\usr\local\HananTestProj.jar");
> > > >
> > > >
> > > > org.apache.flink.api.java.DataSet text =
> > > > (org.apache.flink.api.java.DataSet) envRemote.fromElements(
> > > > "Who's there?",
> > > > "I think I hear them. Stand, ho! Who's there?");
> > > >
> > > > org.apache.flink.api.java.DataSet<Tuple2<String, Integer>>
> > > > wordCounts = text
> > > > .flatMap(new LineSplitter())
> > > > .groupBy(0)
> > > > .sum(1);
> > > >
> > > > wordCounts.print();
> > > > }
> > > >
> > > > public static class LineSplitter implements
> FlatMapFunction<String,
> > > > Tuple2<String, Integer>> {
> > > > public void flatMap(String line, Collector<Tuple2<String,
> > > Integer>>
> > > > out) {
> > > > for (String word : line.split(" ")) {
> > > > out.collect(new Tuple2<String, Integer>(word, 1));
> > > > }
> > > > }
> > > > }
> > > >
> > > > .....
> > > > Program works fine
> > > > 3.Now we are trying to run this program remotely , from windows
> machine
> > > > when the first row looks differently
> > > > ExecutionEnvironment envRemote
> > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4", 6123,
> > > > "C:\\HananTestProj.jar");
> > > > when 1.2.3.4 is IP address of fink machine
> > > >
> > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be reached bla bla
> > bla
> > > >
> > > > 5.in flink configuration we found a following line
> > > jobmanager.rpc.address:
> > > > localhost
> > > > Flink cant be started with any other value (hostname/ipaddress )
> except
> > > the
> > > > localhost
> > > >
> > > >
> > > > 6.We assume that probably Flink has a critical bug : it cant be
> started
> > > > from remote machine, only locally. Are we right? Are we wrong? Should
> > we
> > > > fill JIRA ?
> > > > Maybe we need somehow to configure Flink differently?
> > > >
> > > > Please advice
> > > > Best regards
> > > >
> > > >
> > > >
> > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > The output of the YARN session should look like this:
> > > > >
> > > > > Flink JobManager is now running on quickstart.cloudera:39956
> > > > > JobManager Web Interface:
> > > > >
> > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > Number of connected TaskManagers changed to 1. Slots available: 1
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <[hidden email]>
> > > wrote:
> > > > >
> > > > > > The only thing I can think of is that you are not using the right
> > > > > host/port
> > > > > > for the JobManager.
> > > > > >
> > > > > > When you start the YARN session, it should print the host where
> the
> > > > > > JobManager runs. You also need to take the port from there, as in
> > > YARN,
> > > > > the
> > > > > > port is usually not 6123. Yarn starts many services on one
> machine,
> > > so
> > > > > the
> > > > > > ports need to be randomized.
> > > > > >
> > > > > > It may be worth adding a YARNExecutionEnvironment at some point,
> > > which
> > > > > > deals with this transparent (starting the YARN cluster,
> connecting
> > to
> > > > the
> > > > > > JobManager).
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Hello.
> > > > > > > Let me clarify the situation.
> > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We connected it to
> > HDFS
> > > > > > 2.7.1.
> > > > > > > 2. Locally, our program is working: once we run flink as
> > > > > > ./start-local.sh,
> > > > > > > we are able to connect and run the createRemoteEnvironment and
> > > > Execute
> > > > > > > methods.
> > > > > > > 3.Due to our architecture and basic Flink feature we want to
> > invoke
> > > > > this
> > > > > > > functionality REMOTELY , when our Java code is calling the
> Flink
> > > > > methods
> > > > > > > from another server.
> > > > > > > 4.We tried both
> > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > 6123, "TestProj.jar"); and
> > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is definitely
> not
> > > > right
> > > > > > > since
> > > > > > > it should be an IP address) - it crash on the "cant reach
> > > JobManager"
> > > > > > > error.
> > > > > > >
> > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > 1.Somehow we need to configure flink to accept the connections
> > from
> > > > the
> > > > > > > remote machine
> > > > > > > 2.Flink has a critical showstopper bug that jeopardizing a
> whole
> > > > > decision
> > > > > > > to use this technology.
> > > > > > >
> > > > > > > Please advise us how we should advance.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > in the exception you've posted earlier, you can see the
> > following
> > > > > root
> > > > > > > > cause:
> > > > > > > >
> > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found for:
> > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > :6123/),
> > > > > > > > Path(/user/jobmanager)]
> > > > > > > >
> > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL:6123/"
> usually
> > > > looks
> > > > > > like
> > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it seems that you
> > are
> > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > as the server hostname (or ip).
> > > > > > > > Can you pass the correct hostname when you call
> > > > ExecutionEnvironment.
> > > > > > > > createRemoteEnvironment().
> > > > > > > >
> > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi
> > > > > > > > > I'm currently using flink 0.9.0 which by maven support
> > Hadoop
> > > 1
> > > > .
> > > > > > > > > By using flink-clients-0.7.0-hadoop2-incubating.jar with
> > > > > > > executePlan(Plan
> > > > > > > > > p) method instead, I'm getting the same exception
> > > > > > > > >
> > > > > > > > > Hanan
> > > > > > > > >
> > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > 1. I have restarted Flink service via
> stop/start-loval.sh -
> > > it
> > > > > have
> > > > > > > > been
> > > > > > > > > > restarted successfully ,no errors in log folder
> > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > >
> > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> org.apache.flink.client.program.ProgramInvocationException:
> > > > > Failed
> > > > > > to
> > > > > > > > > > resolve JobManager
> > > > > > > > > > at
> > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > at
> > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > at
> > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL:6123/user/jobmanager
> not
> > > > > > > reachable.
> > > > > > > > > > Please make sure that the JobManager is running and its
> > port
> > > is
> > > > > > > > > reachable.
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > at
> > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > ... 7 more
> > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found for:
> > > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > :6123/),
> > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > at
> > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > >
> > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > at
> > > > > > > >
> > > > >
> > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > at
> > > > > > >
> > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > at
> > > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > at
> > > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > at
> akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > at
> akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > at
> > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > at
> > > > > > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > at
> > > > > > >
> > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger <
> > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> I guess you are getting an entire exception after the
> > > > > > > > "org.apache.flink
> > > > > > > > > >> .client.program.ProgramInvocationException: Failed to
> > > > > > > > > >> resolve JobManager".
> > > > > > > > > >> Can you post it here to help us understanding the issue?
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey Sapozhnikov <
> > > > > > > > > [hidden email]>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hello all.
> > > > > > > > > >> >
> > > > > > > > > >> > Some clarification: locally everything works great.
> > > > > > > > > >> > However once we run our Flink on remote linux machine
> > and
> > > > try
> > > > > to
> > > > > > > run
> > > > > > > > > the
> > > > > > > > > >> > client program from our machine, using create remote
> > > > > > environment-
> > > > > > > > > Flink
> > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > >> >
> > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > If you start the job via the "bin/flink" script,
> then
> > > > simply
> > > > > > use
> > > > > > > > > >> > > "ExecutionEnvironment.getExecutionEnvironment()"
> > rather
> > > > then
> > > > > > > > > creating
> > > > > > > > > >> a
> > > > > > > > > >> > > remote environment manually.
> > > > > > > > > >> > >
> > > > > > > > > >> > > That way, hosts and ports are configured
> > automatically.
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger <
> > > > > > > > > [hidden email]>
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > >> Hi,
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> Which values did you use for FLINK_SERVER_URL and
> > > > > FLINK_PORT?
> > > > > > > > > >> > >> Every time you deploy Flink on YARN, the host and
> > port
> > > > > > change,
> > > > > > > > > >> because
> > > > > > > > > >> > the
> > > > > > > > > >> > >> JobManager is started on a different YARN
> container.
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> > Hello All
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > When using Eclipse IDE to submit Flink to Yarn
> > single
> > > > > node
> > > > > > > > > cluster
> > > > > > > > > >> I'm
> > > > > > > > > >> > >> > getting :
> > > > > > > > > >> > >> >
> > > > > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > >> Failed to
> > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The Jar copy a file from one location in Hdfs to
> > > > another
> > > > > > and
> > > > > > > > > works
> > > > > > > > > >> > fine
> > > > > > > > > >> > >> > while executed locally on the single node Yarn
> > > cluster
> > > > -
> > > > > > > > > >> > >> > bin/flink run -c Test ./examples/MyJar.jar
> > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > >> > >> > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > >> > >> >
> > > > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > >> > >> >
> > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Please advise,
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >>
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Regards*
> >
> > *Alexey Sapozhnikov*
> > CTO& Co-Founder
> > Scalabillit Inc
> > Aba Even 10-C, Herzelia, Israel
> > M : +972-52-2363823
> > E : [hidden email]
> > W : http://www.scalabill.it
> > YT - https://youtu.be/9Rj309PTOFA
> > Map:http://mapta.gs/Scalabillit
> > Revolutionizing Proof-of-Concept
> >
>

--

*Regards*

*Alexey Sapozhnikov*
CTO& Co-Founder
Scalabillit Inc
Aba Even 10-C, Herzelia, Israel
M : +972-52-2363823
E : [hidden email]
W : http://www.scalabill.it
YT - https://youtu.be/9Rj309PTOFA
Map:http://mapta.gs/Scalabillit
Revolutionizing Proof-of-Concept

Robert Metzger

Re: Apache Flink:ProgramInvocationException on Yarn

From this blog post, it seems that this hostname is not resolvable:
https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
Can you easily activate a DNS server in the VPC?

0.0.0.0 is not supported because of some requirements of the Akka framework.
But you should be able to use the host's IP address.
Can you try that as well?

On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <[hidden email]>
wrote:

> Here is the exception from the moment we tried to put in
> jobmanager.rpc.address the hostname of the machine which is ip-172-36-98
> looks like it doesnt recognize this address.
> Why it doesnt support "0.0.0.0"????
>
> 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
> -
>
> --------------------------------------------------------------------------------
> 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
> - Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf
> 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> - Security is not enabled. Starting non-authenticated JobManager.
> 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> - Starting JobManager
> 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
> - Starting JobManager actor system at ip-172-36-98:6123.
> 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
> - Failed to run JobManager.
> java.lang.Exception: Could not create JobManager actor system
> at
>
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> at
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> at
> org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> @
> 21,1
> 71%
>
>
> On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <[hidden email]>
> wrote:
>
> > How is Flink crashing when you start it on the Linux machine in Amazon?
> >
> > Can you post the exception here?
> >
> > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <[hidden email]
> >
> > wrote:
> >
> > > Hello Stephan.
> > >
> > > We run this Linux machine on Amazon, which I predict, most of the
> people
> > > will do.
> > > We tried to put "0.0.0.0" or Public IP of the machine- Flink crashes on
> > > start, it doesnt recognize himself.
> > > It is very strange that it doesnt work with 0.0.0.0- basically this is
> a
> > > way in Java to make it being seen widely.
> > > We tried to put there the hostname - what you get from "hostname"
> > command.
> > > It crashes.
> > > It works only with "localhost" and works only locally
> > > So what you suggest we will put there so the remote client could
> connect.
> > > ?
> > >
> > > Best regards
> > >
> > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <[hidden email]>
> wrote:
> > >
> > > > Do you start Flink via YARN? In that case the
> "jobmanager.rpc.address"
> > is
> > > > not used, because YARN assigns containers/nodes.
> > > >
> > > > If you start Flink in "standalone" mode, this should be the address
> of
> > > the
> > > > node that runs the JobManager. It will be used as the host/IP that
> > Flink
> > > > binds to. The same host should then be used in the RemoteEnvironment.
> > > >
> > > > Stephan
> > > >
> > > >
> > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <[hidden email]>
> > wrote:
> > > >
> > > > > Hello all.
> > > > >
> > > > > Firstly- thank you for your valuable advices.
> > > > > We did some very fine tuned pinpoint test and comes to following
> > > > > conclusions
> > > > >
> > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > 2.Once we copy our Java client program directy to the machine and
> run
> > > it
> > > > > directly there it worked very good
> > > > > The program is
> > > > >
> > > > > .....
> > > > >
> > > > > ExecutionEnvironment envRemote
> > > > > =ExecutionEnvironment.createRemoteEnvironment("localhost", 6123,
> > > > > "\usr\local\HananTestProj.jar");
> > > > >
> > > > >
> > > > > org.apache.flink.api.java.DataSet text =
> > > > > (org.apache.flink.api.java.DataSet) envRemote.fromElements(
> > > > > "Who's there?",
> > > > > "I think I hear them. Stand, ho! Who's there?");
> > > > >
> > > > > org.apache.flink.api.java.DataSet<Tuple2<String, Integer>>
> > > > > wordCounts = text
> > > > > .flatMap(new LineSplitter())
> > > > > .groupBy(0)
> > > > > .sum(1);
> > > > >
> > > > > wordCounts.print();
> > > > > }
> > > > >
> > > > > public static class LineSplitter implements
> > FlatMapFunction<String,
> > > > > Tuple2<String, Integer>> {
> > > > > public void flatMap(String line, Collector<Tuple2<String,
> > > > Integer>>
> > > > > out) {
> > > > > for (String word : line.split(" ")) {
> > > > > out.collect(new Tuple2<String, Integer>(word, 1));
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > .....
> > > > > Program works fine
> > > > > 3.Now we are trying to run this program remotely , from windows
> > machine
> > > > > when the first row looks differently
> > > > > ExecutionEnvironment envRemote
> > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4", 6123,
> > > > > "C:\\HananTestProj.jar");
> > > > > when 1.2.3.4 is IP address of fink machine
> > > > >
> > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be reached bla
> bla
> > > bla
> > > > >
> > > > > 5.in flink configuration we found a following line
> > > > jobmanager.rpc.address:
> > > > > localhost
> > > > > Flink cant be started with any other value (hostname/ipaddress )
> > except
> > > > the
> > > > > localhost
> > > > >
> > > > >
> > > > > 6.We assume that probably Flink has a critical bug : it cant be
> > started
> > > > > from remote machine, only locally. Are we right? Are we wrong?
> Should
> > > we
> > > > > fill JIRA ?
> > > > > Maybe we need somehow to configure Flink differently?
> > > > >
> > > > > Please advice
> > > > > Best regards
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > The output of the YARN session should look like this:
> > > > > >
> > > > > > Flink JobManager is now running on quickstart.cloudera:39956
> > > > > > JobManager Web Interface:
> > > > > >
> > > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > Number of connected TaskManagers changed to 1. Slots available: 1
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > > > The only thing I can think of is that you are not using the
> right
> > > > > > host/port
> > > > > > > for the JobManager.
> > > > > > >
> > > > > > > When you start the YARN session, it should print the host where
> > the
> > > > > > > JobManager runs. You also need to take the port from there, as
> in
> > > > YARN,
> > > > > > the
> > > > > > > port is usually not 6123. Yarn starts many services on one
> > machine,
> > > > so
> > > > > > the
> > > > > > > ports need to be randomized.
> > > > > > >
> > > > > > > It may be worth adding a YARNExecutionEnvironment at some
> point,
> > > > which
> > > > > > > deals with this transparent (starting the YARN cluster,
> > connecting
> > > to
> > > > > the
> > > > > > > JobManager).
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > [hidden email]>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello.
> > > > > > > > Let me clarify the situation.
> > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We connected it
> to
> > > HDFS
> > > > > > > 2.7.1.
> > > > > > > > 2. Locally, our program is working: once we run flink as
> > > > > > > ./start-local.sh,
> > > > > > > > we are able to connect and run the createRemoteEnvironment
> and
> > > > > Execute
> > > > > > > > methods.
> > > > > > > > 3.Due to our architecture and basic Flink feature we want to
> > > invoke
> > > > > > this
> > > > > > > > functionality REMOTELY , when our Java code is calling the
> > Flink
> > > > > > methods
> > > > > > > > from another server.
> > > > > > > > 4.We tried both
> > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > 6123, "TestProj.jar"); and
> > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is definitely
> > not
> > > > > right
> > > > > > > > since
> > > > > > > > it should be an IP address) - it crash on the "cant reach
> > > > JobManager"
> > > > > > > > error.
> > > > > > > >
> > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > 1.Somehow we need to configure flink to accept the
> connections
> > > from
> > > > > the
> > > > > > > > remote machine
> > > > > > > > 2.Flink has a critical showstopper bug that jeopardizing a
> > whole
> > > > > > decision
> > > > > > > > to use this technology.
> > > > > > > >
> > > > > > > > Please advise us how we should advance.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > in the exception you've posted earlier, you can see the
> > > following
> > > > > > root
> > > > > > > > > cause:
> > > > > > > > >
> > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found for:
> > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > :6123/),
> > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > >
> > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL:6123/"
> > usually
> > > > > looks
> > > > > > > like
> > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it seems that
> you
> > > are
> > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > as the server hostname (or ip).
> > > > > > > > > Can you pass the correct hostname when you call
> > > > > ExecutionEnvironment.
> > > > > > > > > createRemoteEnvironment().
> > > > > > > > >
> > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > > I'm currently using flink 0.9.0 which by maven support
> > > Hadoop
> > > > 1
> > > > > .
> > > > > > > > > > By using flink-clients-0.7.0-hadoop2-incubating.jar with
> > > > > > > > executePlan(Plan
> > > > > > > > > > p) method instead, I'm getting the same exception
> > > > > > > > > >
> > > > > > > > > > Hanan
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi
> > > > > > > > > > >
> > > > > > > > > > > 1. I have restarted Flink service via
> > stop/start-loval.sh -
> > > > it
> > > > > > have
> > > > > > > > > been
> > > > > > > > > > > restarted successfully ,no errors in log folder
> > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > >
> > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > Failed
> > > > > > > to
> > > > > > > > > > > resolve JobManager
> > > > > > > > > > > at
> > > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > at
> > > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > at
> > > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL:6123/user/jobmanager
> > not
> > > > > > > > reachable.
> > > > > > > > > > > Please make sure that the JobManager is running and its
> > > port
> > > > is
> > > > > > > > > > reachable.
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > at
> > > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > ... 7 more
> > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found
> for:
> > > > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > :6123/),
> > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > at
> > > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > >
> > > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > at
> > > > > > > >
> > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > at
> > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > at
> > > > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > at
> > > > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > at
> > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > at
> > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > at
> akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > at
> > > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > at
> akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > at
> akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > at
> > > > > > >
> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > at
> > > > > > > >
> > > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger <
> > > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I guess you are getting an entire exception after the
> > > > > > > > > "org.apache.flink
> > > > > > > > > > >> .client.program.ProgramInvocationException: Failed to
> > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > >> Can you post it here to help us understanding the
> issue?
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey Sapozhnikov <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hello all.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Some clarification: locally everything works great.
> > > > > > > > > > >> > However once we run our Flink on remote linux
> machine
> > > and
> > > > > try
> > > > > > to
> > > > > > > > run
> > > > > > > > > > the
> > > > > > > > > > >> > client program from our machine, using create remote
> > > > > > > environment-
> > > > > > > > > > Flink
> > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > If you start the job via the "bin/flink" script,
> > then
> > > > > simply
> > > > > > > use
> > > > > > > > > > >> > > "ExecutionEnvironment.getExecutionEnvironment()"
> > > rather
> > > > > then
> > > > > > > > > > creating
> > > > > > > > > > >> a
> > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > That way, hosts and ports are configured
> > > automatically.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >> Hi,
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> Which values did you use for FLINK_SERVER_URL and
> > > > > > FLINK_PORT?
> > > > > > > > > > >> > >> Every time you deploy Flink on YARN, the host and
> > > port
> > > > > > > change,
> > > > > > > > > > >> because
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >> JobManager is started on a different YARN
> > container.
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > When using Eclipse IDE to submit Flink to Yarn
> > > single
> > > > > > node
> > > > > > > > > > cluster
> > > > > > > > > > >> I'm
> > > > > > > > > > >> > >> > getting :
> > > > > > > > > > >> > >> >
> > > > > > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > >> Failed to
> > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The Jar copy a file from one location in Hdfs
> to
> > > > > another
> > > > > > > and
> > > > > > > > > > works
> > > > > > > > > > >> > fine
> > > > > > > > > > >> > >> > while executed locally on the single node Yarn
> > > > cluster
> > > > > -
> > > > > > > > > > >> > >> > bin/flink run -c Test ./examples/MyJar.jar
> > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > > >> > >> > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > >> > >> >
> > > > > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > >> > >> >
> > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Regards*
> > >
> > > *Alexey Sapozhnikov*
> > > CTO& Co-Founder
> > > Scalabillit Inc
> > > Aba Even 10-C, Herzelia, Israel
> > > M : +972-52-2363823
> > > E : [hidden email]
> > > W : http://www.scalabill.it
> > > YT - https://youtu.be/9Rj309PTOFA
> > > Map:http://mapta.gs/Scalabillit
> > > Revolutionizing Proof-of-Concept
> > >
> >
>
>
>
> --
>
> *Regards*
>
> *Alexey Sapozhnikov*
> CTO& Co-Founder
> Scalabillit Inc
> Aba Even 10-C, Herzelia, Israel
> M : +972-52-2363823
> E : [hidden email]
> W : http://www.scalabill.it
> YT - https://youtu.be/9Rj309PTOFA
> Map:http://mapta.gs/Scalabillit
> Revolutionizing Proof-of-Concept
>

Stephan Ewen

Re: Apache Flink:ProgramInvocationException on Yarn

In reply to this post by Alexey Sapozhnikov

Weird, the root cause seems to be "java.net.UnknownHostException:
ip-172-36-98: unknown error"

Flink does not do anything more special than
"InetAddress.getByName(hostname)".

Is it that you can either not resolve the hostname "ip-172-36-98" (maybe
add the fully qualified domain name), or is there another DNS lookup error
in the JVM setup?

If the default Java DNS provider has problems in that setup, you could try
to set another DNS provider...

Stephan

On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <[hidden email]>
wrote:

Alexey Sapozhnikov

Re: Apache Flink:ProgramInvocationException on Yarn

In reply to this post by Robert Metzger

Hi.
First off - many thanks for your efforts and prompt help.
We will try to find how to do it with DNS server on VPC.
however, absence of "0.0.0.0" is definitely a huge bug - just think about
the current situation : if I dont have a VPC, I cant invoke the Flink
functionality remotely in Amazon.
We will clarify the VPC situation and I will write my impression of what we
get.

Thanks, guys

On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <[hidden email]> wrote:

> From this blog post, it seems that this hostname is not resolvable:
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> Can you easily activate a DNS server in the VPC?
>
> 0.0.0.0 is not supported because of some requirements of the Akka
> framework.
> But you should be able to use the host's IP address.
> Can you try that as well?
>
>
>
>
> On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <[hidden email]>
> wrote:
>
> > Here is the exception from the moment we tried to put in
> > jobmanager.rpc.address the hostname of the machine which is ip-172-36-98
> > looks like it doesnt recognize this address.
> > Why it doesnt support "0.0.0.0"????
> >
> > 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
> > -
> >
> >
> --------------------------------------------------------------------------------
> > 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
> > - Loading configuration from /usr/local/test/flink-0.9.0/bin/../conf
> > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > - Security is not enabled. Starting non-authenticated JobManager.
> > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > - Starting JobManager
> > 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
> > - Starting JobManager actor system at ip-172-36-98:6123.
> > 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
> > - Failed to run JobManager.
> > java.lang.Exception: Could not create JobManager actor system
> > at
> >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > at
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > at
> > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
> > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > @
> > 21,1
> > 71%
> >
> >
> > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <[hidden email]>
> > wrote:
> >
> > > How is Flink crashing when you start it on the Linux machine in Amazon?
> > >
> > > Can you post the exception here?
> > >
> > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Hello Stephan.
> > > >
> > > > We run this Linux machine on Amazon, which I predict, most of the
> > people
> > > > will do.
> > > > We tried to put "0.0.0.0" or Public IP of the machine- Flink crashes
> on
> > > > start, it doesnt recognize himself.
> > > > It is very strange that it doesnt work with 0.0.0.0- basically this
> is
> > a
> > > > way in Java to make it being seen widely.
> > > > We tried to put there the hostname - what you get from "hostname"
> > > command.
> > > > It crashes.
> > > > It works only with "localhost" and works only locally
> > > > So what you suggest we will put there so the remote client could
> > connect.
> > > > ?
> > > >
> > > > Best regards
> > > >
> > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <[hidden email]>
> > wrote:
> > > >
> > > > > Do you start Flink via YARN? In that case the
> > "jobmanager.rpc.address"
> > > is
> > > > > not used, because YARN assigns containers/nodes.
> > > > >
> > > > > If you start Flink in "standalone" mode, this should be the address
> > of
> > > > the
> > > > > node that runs the JobManager. It will be used as the host/IP that
> > > Flink
> > > > > binds to. The same host should then be used in the
> RemoteEnvironment.
> > > > >
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hello all.
> > > > > >
> > > > > > Firstly- thank you for your valuable advices.
> > > > > > We did some very fine tuned pinpoint test and comes to following
> > > > > > conclusions
> > > > > >
> > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > 2.Once we copy our Java client program directy to the machine and
> > run
> > > > it
> > > > > > directly there it worked very good
> > > > > > The program is
> > > > > >
> > > > > > .....
> > > > > >
> > > > > > ExecutionEnvironment envRemote
> > > > > > =ExecutionEnvironment.createRemoteEnvironment("localhost", 6123,
> > > > > > "\usr\local\HananTestProj.jar");
> > > > > >
> > > > > >
> > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > (org.apache.flink.api.java.DataSet) envRemote.fromElements(
> > > > > > "Who's there?",
> > > > > > "I think I hear them. Stand, ho! Who's there?");
> > > > > >
> > > > > > org.apache.flink.api.java.DataSet<Tuple2<String, Integer>>
> > > > > > wordCounts = text
> > > > > > .flatMap(new LineSplitter())
> > > > > > .groupBy(0)
> > > > > > .sum(1);
> > > > > >
> > > > > > wordCounts.print();
> > > > > > }
> > > > > >
> > > > > > public static class LineSplitter implements
> > > FlatMapFunction<String,
> > > > > > Tuple2<String, Integer>> {
> > > > > > public void flatMap(String line, Collector<Tuple2<String,
> > > > > Integer>>
> > > > > > out) {
> > > > > > for (String word : line.split(" ")) {
> > > > > > out.collect(new Tuple2<String, Integer>(word, 1));
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > .....
> > > > > > Program works fine
> > > > > > 3.Now we are trying to run this program remotely , from windows
> > > machine
> > > > > > when the first row looks differently
> > > > > > ExecutionEnvironment envRemote
> > > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4", 6123,
> > > > > > "C:\\HananTestProj.jar");
> > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > >
> > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be reached bla
> > bla
> > > > bla
> > > > > >
> > > > > > 5.in flink configuration we found a following line
> > > > > jobmanager.rpc.address:
> > > > > > localhost
> > > > > > Flink cant be started with any other value (hostname/ipaddress )
> > > except
> > > > > the
> > > > > > localhost
> > > > > >
> > > > > >
> > > > > > 6.We assume that probably Flink has a critical bug : it cant be
> > > started
> > > > > > from remote machine, only locally. Are we right? Are we wrong?
> > Should
> > > > we
> > > > > > fill JIRA ?
> > > > > > Maybe we need somehow to configure Flink differently?
> > > > > >
> > > > > > Please advice
> > > > > > Best regards
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > The output of the YARN session should look like this:
> > > > > > >
> > > > > > > Flink JobManager is now running on quickstart.cloudera:39956
> > > > > > > JobManager Web Interface:
> > > > > > >
> > > >
> http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > Number of connected TaskManagers changed to 1. Slots
> available: 1
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > The only thing I can think of is that you are not using the
> > right
> > > > > > > host/port
> > > > > > > > for the JobManager.
> > > > > > > >
> > > > > > > > When you start the YARN session, it should print the host
> where
> > > the
> > > > > > > > JobManager runs. You also need to take the port from there,
> as
> > in
> > > > > YARN,
> > > > > > > the
> > > > > > > > port is usually not 6123. Yarn starts many services on one
> > > machine,
> > > > > so
> > > > > > > the
> > > > > > > > ports need to be randomized.
> > > > > > > >
> > > > > > > > It may be worth adding a YARNExecutionEnvironment at some
> > point,
> > > > > which
> > > > > > > > deals with this transparent (starting the YARN cluster,
> > > connecting
> > > > to
> > > > > > the
> > > > > > > > JobManager).
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello.
> > > > > > > > > Let me clarify the situation.
> > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We connected it
> > to
> > > > HDFS
> > > > > > > > 2.7.1.
> > > > > > > > > 2. Locally, our program is working: once we run flink as
> > > > > > > > ./start-local.sh,
> > > > > > > > > we are able to connect and run the createRemoteEnvironment
> > and
> > > > > > Execute
> > > > > > > > > methods.
> > > > > > > > > 3.Due to our architecture and basic Flink feature we want
> to
> > > > invoke
> > > > > > > this
> > > > > > > > > functionality REMOTELY , when our Java code is calling the
> > > Flink
> > > > > > > methods
> > > > > > > > > from another server.
> > > > > > > > > 4.We tried both
> > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is
> definitely
> > > not
> > > > > > right
> > > > > > > > > since
> > > > > > > > > it should be an IP address) - it crash on the "cant reach
> > > > > JobManager"
> > > > > > > > > error.
> > > > > > > > >
> > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > 1.Somehow we need to configure flink to accept the
> > connections
> > > > from
> > > > > > the
> > > > > > > > > remote machine
> > > > > > > > > 2.Flink has a critical showstopper bug that jeopardizing a
> > > whole
> > > > > > > decision
> > > > > > > > > to use this technology.
> > > > > > > > >
> > > > > > > > > Please advise us how we should advance.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > in the exception you've posted earlier, you can see the
> > > > following
> > > > > > > root
> > > > > > > > > > cause:
> > > > > > > > > >
> > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found for:
> > > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > :6123/),
> > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > >
> > > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL:6123/"
> > > usually
> > > > > > looks
> > > > > > > > like
> > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it seems that
> > you
> > > > are
> > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > Can you pass the correct hostname when you call
> > > > > > ExecutionEnvironment.
> > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi
> > > > > > > > > > > I'm currently using flink 0.9.0 which by maven support
> > > > Hadoop
> > > > > 1
> > > > > > .
> > > > > > > > > > > By using flink-clients-0.7.0-hadoop2-incubating.jar
> with
> > > > > > > > > executePlan(Plan
> > > > > > > > > > > p) method instead, I'm getting the same exception
> > > > > > > > > > >
> > > > > > > > > > > Hanan
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi
> > > > > > > > > > > >
> > > > > > > > > > > > 1. I have restarted Flink service via
> > > stop/start-loval.sh -
> > > > > it
> > > > > > > have
> > > > > > > > > > been
> > > > > > > > > > > > restarted successfully ,no errors in log folder
> > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > >
> > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > Failed
> > > > > > > > to
> > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > at
> > > > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > at
> > > > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > at
> > > > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> :6123/user/jobmanager
> > > not
> > > > > > > > > reachable.
> > > > > > > > > > > > Please make sure that the JobManager is running and
> its
> > > > port
> > > > > is
> > > > > > > > > > > reachable.
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > at
> > > > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found
> > for:
> > > > > > > > > > > >
> ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > :6123/),
> > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > at
> > > > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > >
> > > > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > >
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > at
> > > > > > > > >
> > > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > at
> > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > at
> > > > > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > at
> > > > > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > at
> > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > at
> > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > at
> > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > at
> > > > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > at
> akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > at
> > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > at
> > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > at
> > > > > > > >
> > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > at
> > > > > > > > >
> > > > >
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> I guess you are getting an entire exception after
> the
> > > > > > > > > > "org.apache.flink
> > > > > > > > > > > >> .client.program.ProgramInvocationException: Failed
> to
> > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > >> Can you post it here to help us understanding the
> > issue?
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey Sapozhnikov
> <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Some clarification: locally everything works
> great.
> > > > > > > > > > > >> > However once we run our Flink on remote linux
> > machine
> > > > and
> > > > > > try
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > the
> > > > > > > > > > > >> > client program from our machine, using create
> remote
> > > > > > > > environment-
> > > > > > > > > > > Flink
> > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > If you start the job via the "bin/flink" script,
> > > then
> > > > > > simply
> > > > > > > > use
> > > > > > > > > > > >> > > "ExecutionEnvironment.getExecutionEnvironment()"
> > > > rather
> > > > > > then
> > > > > > > > > > > creating
> > > > > > > > > > > >> a
> > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > That way, hosts and ports are configured
> > > > automatically.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger
> <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > >> > >>
> > > > > > > > > > > >> > >> Which values did you use for FLINK_SERVER_URL
> and
> > > > > > > FLINK_PORT?
> > > > > > > > > > > >> > >> Every time you deploy Flink on YARN, the host
> and
> > > > port
> > > > > > > > change,
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > >> JobManager is started on a different YARN
> > > container.
> > > > > > > > > > > >> > >>
> > > > > > > > > > > >> > >>
> > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> > >>
> > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > When using Eclipse IDE to submit Flink to
> Yarn
> > > > single
> > > > > > > node
> > > > > > > > > > > cluster
> > > > > > > > > > > >> I'm
> > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > >> > >> >
> > > > > > > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > >> Failed to
> > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > The Jar copy a file from one location in Hdfs
> > to
> > > > > > another
> > > > > > > > and
> > > > > > > > > > > works
> > > > > > > > > > > >> > fine
> > > > > > > > > > > >> > >> > while executed locally on the single node
> Yarn
> > > > > cluster
> > > > > > -
> > > > > > > > > > > >> > >> > bin/flink run -c Test ./examples/MyJar.jar
> > > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > > > >> > >> > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > >> > >> >
> > > > > > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > >> > >> >
> > > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > >> > >> >
> > > > > > > > > > > >> > >>
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Regards*
> > > >
> > > > *Alexey Sapozhnikov*
> > > > CTO& Co-Founder
> > > > Scalabillit Inc
> > > > Aba Even 10-C, Herzelia, Israel
> > > > M : +972-52-2363823
> > > > E : [hidden email]
> > > > W : http://www.scalabill.it
> > > > YT - https://youtu.be/9Rj309PTOFA
> > > > Map:http://mapta.gs/Scalabillit
> > > > Revolutionizing Proof-of-Concept
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Regards*
> >
> > *Alexey Sapozhnikov*
> > CTO& Co-Founder
> > Scalabillit Inc
> > Aba Even 10-C, Herzelia, Israel
> > M : +972-52-2363823
> > E : [hidden email]
> > W : http://www.scalabill.it
> > YT - https://youtu.be/9Rj309PTOFA
> > Map:http://mapta.gs/Scalabillit
> > Revolutionizing Proof-of-Concept
> >
>

Stephan Ewen

Re: Apache Flink:ProgramInvocationException on Yarn

Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes
annoying, but I have not found a good way around this.
The problem is that the address to bind to an the address used by others to
send messages to the node is the same. (
https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY)

All of that aside, we have been using Flink on Cloud Platforms GCE, Amazon,
and it worked.

Can you try to use the internal hostname, or the fully qualified external
hostname?

On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov <[hidden email]>
wrote:

> Hi.
> First off - many thanks for your efforts and prompt help.
> We will try to find how to do it with DNS server on VPC.
> however, absence of "0.0.0.0" is definitely a huge bug - just think about
> the current situation : if I dont have a VPC, I cant invoke the Flink
> functionality remotely in Amazon.
> We will clarify the VPC situation and I will write my impression of what we
> get.
>
> Thanks, guys
>
> On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <[hidden email]>
> wrote:
>
> > From this blog post, it seems that this hostname is not resolvable:
> >
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> > Can you easily activate a DNS server in the VPC?
> >
> > 0.0.0.0 is not supported because of some requirements of the Akka
> > framework.
> > But you should be able to use the host's IP address.
> > Can you try that as well?
> >
> >
> >
> >
> > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <[hidden email]
> >
> > wrote:
> >
> > > Here is the exception from the moment we tried to put in
> > > jobmanager.rpc.address the hostname of the machine which is
> ip-172-36-98
> > > looks like it doesnt recognize this address.
> > > Why it doesnt support "0.0.0.0"????
> > >
> > > 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > -
> > >
> > >
> >
> --------------------------------------------------------------------------------
> > > 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > - Loading configuration from
> /usr/local/test/flink-0.9.0/bin/../conf
> > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > - Security is not enabled. Starting non-authenticated JobManager.
> > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > - Starting JobManager
> > > 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > - Starting JobManager actor system at ip-172-36-98:6123.
> > > 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
> > > - Failed to run JobManager.
> > > java.lang.Exception: Could not create JobManager actor system
> > > at
> > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > > at
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > > at
> > > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > > Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
> > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > > at
> java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > > @
> > > 21,1
> > > 71%
> > >
> > >
> > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <[hidden email]>
> > > wrote:
> > >
> > > > How is Flink crashing when you start it on the Linux machine in
> Amazon?
> > > >
> > > > Can you post the exception here?
> > > >
> > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Hello Stephan.
> > > > >
> > > > > We run this Linux machine on Amazon, which I predict, most of the
> > > people
> > > > > will do.
> > > > > We tried to put "0.0.0.0" or Public IP of the machine- Flink
> crashes
> > on
> > > > > start, it doesnt recognize himself.
> > > > > It is very strange that it doesnt work with 0.0.0.0- basically this
> > is
> > > a
> > > > > way in Java to make it being seen widely.
> > > > > We tried to put there the hostname - what you get from "hostname"
> > > > command.
> > > > > It crashes.
> > > > > It works only with "localhost" and works only locally
> > > > > So what you suggest we will put there so the remote client could
> > > connect.
> > > > > ?
> > > > >
> > > > > Best regards
> > > > >
> > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Do you start Flink via YARN? In that case the
> > > "jobmanager.rpc.address"
> > > > is
> > > > > > not used, because YARN assigns containers/nodes.
> > > > > >
> > > > > > If you start Flink in "standalone" mode, this should be the
> address
> > > of
> > > > > the
> > > > > > node that runs the JobManager. It will be used as the host/IP
> that
> > > > Flink
> > > > > > binds to. The same host should then be used in the
> > RemoteEnvironment.
> > > > > >
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > > > Hello all.
> > > > > > >
> > > > > > > Firstly- thank you for your valuable advices.
> > > > > > > We did some very fine tuned pinpoint test and comes to
> following
> > > > > > > conclusions
> > > > > > >
> > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > > 2.Once we copy our Java client program directy to the machine
> and
> > > run
> > > > > it
> > > > > > > directly there it worked very good
> > > > > > > The program is
> > > > > > >
> > > > > > > .....
> > > > > > >
> > > > > > > ExecutionEnvironment envRemote
> > > > > > > =ExecutionEnvironment.createRemoteEnvironment("localhost",
> 6123,
> > > > > > > "\usr\local\HananTestProj.jar");
> > > > > > >
> > > > > > >
> > > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > > (org.apache.flink.api.java.DataSet) envRemote.fromElements(
> > > > > > > "Who's there?",
> > > > > > > "I think I hear them. Stand, ho! Who's there?");
> > > > > > >
> > > > > > > org.apache.flink.api.java.DataSet<Tuple2<String,
> Integer>>
> > > > > > > wordCounts = text
> > > > > > > .flatMap(new LineSplitter())
> > > > > > > .groupBy(0)
> > > > > > > .sum(1);
> > > > > > >
> > > > > > > wordCounts.print();
> > > > > > > }
> > > > > > >
> > > > > > > public static class LineSplitter implements
> > > > FlatMapFunction<String,
> > > > > > > Tuple2<String, Integer>> {
> > > > > > > public void flatMap(String line,
> Collector<Tuple2<String,
> > > > > > Integer>>
> > > > > > > out) {
> > > > > > > for (String word : line.split(" ")) {
> > > > > > > out.collect(new Tuple2<String, Integer>(word,
> 1));
> > > > > > > }
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > .....
> > > > > > > Program works fine
> > > > > > > 3.Now we are trying to run this program remotely , from windows
> > > > machine
> > > > > > > when the first row looks differently
> > > > > > > ExecutionEnvironment envRemote
> > > > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4", 6123,
> > > > > > > "C:\\HananTestProj.jar");
> > > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > > >
> > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be reached
> bla
> > > bla
> > > > > bla
> > > > > > >
> > > > > > > 5.in flink configuration we found a following line
> > > > > > jobmanager.rpc.address:
> > > > > > > localhost
> > > > > > > Flink cant be started with any other value (hostname/ipaddress
> )
> > > > except
> > > > > > the
> > > > > > > localhost
> > > > > > >
> > > > > > >
> > > > > > > 6.We assume that probably Flink has a critical bug : it cant be
> > > > started
> > > > > > > from remote machine, only locally. Are we right? Are we wrong?
> > > Should
> > > > > we
> > > > > > > fill JIRA ?
> > > > > > > Maybe we need somehow to configure Flink differently?
> > > > > > >
> > > > > > > Please advice
> > > > > > > Best regards
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > The output of the YARN session should look like this:
> > > > > > > >
> > > > > > > > Flink JobManager is now running on quickstart.cloudera:39956
> > > > > > > > JobManager Web Interface:
> > > > > > > >
> > > > >
> > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > > Number of connected TaskManagers changed to 1. Slots
> > available: 1
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > The only thing I can think of is that you are not using the
> > > right
> > > > > > > > host/port
> > > > > > > > > for the JobManager.
> > > > > > > > >
> > > > > > > > > When you start the YARN session, it should print the host
> > where
> > > > the
> > > > > > > > > JobManager runs. You also need to take the port from there,
> > as
> > > in
> > > > > > YARN,
> > > > > > > > the
> > > > > > > > > port is usually not 6123. Yarn starts many services on one
> > > > machine,
> > > > > > so
> > > > > > > > the
> > > > > > > > > ports need to be randomized.
> > > > > > > > >
> > > > > > > > > It may be worth adding a YARNExecutionEnvironment at some
> > > point,
> > > > > > which
> > > > > > > > > deals with this transparent (starting the YARN cluster,
> > > > connecting
> > > > > to
> > > > > > > the
> > > > > > > > > JobManager).
> > > > > > > > >
> > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello.
> > > > > > > > > > Let me clarify the situation.
> > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We connected
> it
> > > to
> > > > > HDFS
> > > > > > > > > 2.7.1.
> > > > > > > > > > 2. Locally, our program is working: once we run flink as
> > > > > > > > > ./start-local.sh,
> > > > > > > > > > we are able to connect and run the
> createRemoteEnvironment
> > > and
> > > > > > > Execute
> > > > > > > > > > methods.
> > > > > > > > > > 3.Due to our architecture and basic Flink feature we want
> > to
> > > > > invoke
> > > > > > > > this
> > > > > > > > > > functionality REMOTELY , when our Java code is calling
> the
> > > > Flink
> > > > > > > > methods
> > > > > > > > > > from another server.
> > > > > > > > > > 4.We tried both
> > > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is
> > definitely
> > > > not
> > > > > > > right
> > > > > > > > > > since
> > > > > > > > > > it should be an IP address) - it crash on the "cant reach
> > > > > > JobManager"
> > > > > > > > > > error.
> > > > > > > > > >
> > > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > > 1.Somehow we need to configure flink to accept the
> > > connections
> > > > > from
> > > > > > > the
> > > > > > > > > > remote machine
> > > > > > > > > > 2.Flink has a critical showstopper bug that jeopardizing
> a
> > > > whole
> > > > > > > > decision
> > > > > > > > > > to use this technology.
> > > > > > > > > >
> > > > > > > > > > Please advise us how we should advance.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > in the exception you've posted earlier, you can see the
> > > > > following
> > > > > > > > root
> > > > > > > > > > > cause:
> > > > > > > > > > >
> > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found
> for:
> > > > > > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > :6123/),
> > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > >
> > > > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL:6123/"
> > > > usually
> > > > > > > looks
> > > > > > > > > like
> > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it seems
> that
> > > you
> > > > > are
> > > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > > Can you pass the correct hostname when you call
> > > > > > > ExecutionEnvironment.
> > > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi
> > > > > > > > > > > > I'm currently using flink 0.9.0 which by maven
> support
> > > > > Hadoop
> > > > > > 1
> > > > > > > .
> > > > > > > > > > > > By using flink-clients-0.7.0-hadoop2-incubating.jar
> > with
> > > > > > > > > > executePlan(Plan
> > > > > > > > > > > > p) method instead, I'm getting the same exception
> > > > > > > > > > > >
> > > > > > > > > > > > Hanan
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. I have restarted Flink service via
> > > > stop/start-loval.sh -
> > > > > > it
> > > > > > > > have
> > > > > > > > > > > been
> > > > > > > > > > > > > restarted successfully ,no errors in log folder
> > > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > > >
> > > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > Failed
> > > > > > > > > to
> > > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > > at
> > > > > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > > at
> > > > > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > > at
> > > > > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> > :6123/user/jobmanager
> > > > not
> > > > > > > > > > reachable.
> > > > > > > > > > > > > Please make sure that the JobManager is running and
> > its
> > > > > port
> > > > > > is
> > > > > > > > > > > > reachable.
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > > at
> > > > > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not
> found
> > > for:
> > > > > > > > > > > > >
> > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > :6123/),
> > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > > at
> > > > > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > > >
> > > > > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > > at
> > > > > > > > > >
> > > > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > > at
> > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > > at
> > > > > > > >
> akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > > at
> > > > > > > >
> akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > > at
> > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > > at
> > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > > at
> > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > > at
> > > > > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > > at
> > akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > > at
> > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > > at
> > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > > at
> > > > > > > > >
> > > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > > at
> > > > > > > > > >
> > > > > >
> > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> I guess you are getting an entire exception after
> > the
> > > > > > > > > > > "org.apache.flink
> > > > > > > > > > > > >> .client.program.ProgramInvocationException: Failed
> > to
> > > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > > >> Can you post it here to help us understanding the
> > > issue?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey
> Sapozhnikov
> > <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Some clarification: locally everything works
> > great.
> > > > > > > > > > > > >> > However once we run our Flink on remote linux
> > > machine
> > > > > and
> > > > > > > try
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > > the
> > > > > > > > > > > > >> > client program from our machine, using create
> > remote
> > > > > > > > > environment-
> > > > > > > > > > > > Flink
> > > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > If you start the job via the "bin/flink"
> script,
> > > > then
> > > > > > > simply
> > > > > > > > > use
> > > > > > > > > > > > >> > >
> "ExecutionEnvironment.getExecutionEnvironment()"
> > > > > rather
> > > > > > > then
> > > > > > > > > > > > creating
> > > > > > > > > > > > >> a
> > > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > That way, hosts and ports are configured
> > > > > automatically.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert
> Metzger
> > <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > > >> > >>
> > > > > > > > > > > > >> > >> Which values did you use for FLINK_SERVER_URL
> > and
> > > > > > > > FLINK_PORT?
> > > > > > > > > > > > >> > >> Every time you deploy Flink on YARN, the host
> > and
> > > > > port
> > > > > > > > > change,
> > > > > > > > > > > > >> because
> > > > > > > > > > > > >> > the
> > > > > > > > > > > > >> > >> JobManager is started on a different YARN
> > > > container.
> > > > > > > > > > > > >> > >>
> > > > > > > > > > > > >> > >>
> > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> > >>
> > > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit Flink to
> > Yarn
> > > > > single
> > > > > > > > node
> > > > > > > > > > > > cluster
> > > > > > > > > > > > >> I'm
> > > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > > >> > >> >
> > > > > > > > >
> "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > >> Failed to
> > > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > The Jar copy a file from one location in
> Hdfs
> > > to
> > > > > > > another
> > > > > > > > > and
> > > > > > > > > > > > works
> > > > > > > > > > > > >> > fine
> > > > > > > > > > > > >> > >> > while executed locally on the single node
> > Yarn
> > > > > > cluster
> > > > > > > -
> > > > > > > > > > > > >> > >> > bin/flink run -c Test ./examples/MyJar.jar
> > > > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > > > > >> > >> >
> ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > > >> > >> >
> > > > > > > > >
> envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > > >> > >> >
> > > > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >> > >>
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Regards*
> > > > >
> > > > > *Alexey Sapozhnikov*
> > > > > CTO& Co-Founder
> > > > > Scalabillit Inc
> > > > > Aba Even 10-C, Herzelia, Israel
> > > > > M : +972-52-2363823
> > > > > E : [hidden email]
> > > > > W : http://www.scalabill.it
> > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > Map:http://mapta.gs/Scalabillit
> > > > > Revolutionizing Proof-of-Concept
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Regards*
> > >
> > > *Alexey Sapozhnikov*
> > > CTO& Co-Founder
> > > Scalabillit Inc
> > > Aba Even 10-C, Herzelia, Israel
> > > M : +972-52-2363823
> > > E : [hidden email]
> > > W : http://www.scalabill.it
> > > YT - https://youtu.be/9Rj309PTOFA
> > > Map:http://mapta.gs/Scalabillit
> > > Revolutionizing Proof-of-Concept
> > >
> >
>
>
>
> --
>
> *Regards*
>
> *Alexey Sapozhnikov*
> CTO& Co-Founder
> Scalabillit Inc
> Aba Even 10-C, Herzelia, Israel
> M : +972-52-2363823
> E : [hidden email]
> W : http://www.scalabill.it
> YT - https://youtu.be/9Rj309PTOFA
> Map:http://mapta.gs/Scalabillit
> Revolutionizing Proof-of-Concept
>

Alexey Sapozhnikov

Re: Apache Flink:ProgramInvocationException on Yarn

Fully understand.
1.My suggestion is to drop Akka and take something else, since this issue
is really big
2.Not hostname not the endpoint are not working, clarifying the VPC topic
now.

On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen <[hidden email]> wrote:

> Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes
> annoying, but I have not found a good way around this.
> The problem is that the address to bind to an the address used by others to
> send messages to the node is the same. (
> https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY)
>
>
> All of that aside, we have been using Flink on Cloud Platforms GCE, Amazon,
> and it worked.
>
> Can you try to use the internal hostname, or the fully qualified external
> hostname?
>
>
> On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov <[hidden email]>
> wrote:
>
> > Hi.
> > First off - many thanks for your efforts and prompt help.
> > We will try to find how to do it with DNS server on VPC.
> > however, absence of "0.0.0.0" is definitely a huge bug - just think about
> > the current situation : if I dont have a VPC, I cant invoke the Flink
> > functionality remotely in Amazon.
> > We will clarify the VPC situation and I will write my impression of what
> we
> > get.
> >
> > Thanks, guys
> >
> > On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <[hidden email]>
> > wrote:
> >
> > > From this blog post, it seems that this hostname is not resolvable:
> > >
> >
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> > > Can you easily activate a DNS server in the VPC?
> > >
> > > 0.0.0.0 is not supported because of some requirements of the Akka
> > > framework.
> > > But you should be able to use the host's IP address.
> > > Can you try that as well?
> > >
> > >
> > >
> > >
> > > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Here is the exception from the moment we tried to put in
> > > > jobmanager.rpc.address the hostname of the machine which is
> > ip-172-36-98
> > > > looks like it doesnt recognize this address.
> > > > Why it doesnt support "0.0.0.0"????
> > > >
> > > > 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > -
> > > >
> > > >
> > >
> >
> --------------------------------------------------------------------------------
> > > > 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > - Loading configuration from
> > /usr/local/test/flink-0.9.0/bin/../conf
> > > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > - Security is not enabled. Starting non-authenticated
> JobManager.
> > > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > - Starting JobManager
> > > > 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > - Starting JobManager actor system at ip-172-36-98:6123.
> > > > 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
> > > > - Failed to run JobManager.
> > > > java.lang.Exception: Could not create JobManager actor system
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > > > at
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > > > at
> > > > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > > > Caused by: java.net.UnknownHostException: ip-172-36-98: unknown error
> > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > > > at
> > java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > > > @
> > > >
> 21,1
> > > > 71%
> > > >
> > > >
> > > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > How is Flink crashing when you start it on the Linux machine in
> > Amazon?
> > > > >
> > > > > Can you post the exception here?
> > > > >
> > > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello Stephan.
> > > > > >
> > > > > > We run this Linux machine on Amazon, which I predict, most of the
> > > > people
> > > > > > will do.
> > > > > > We tried to put "0.0.0.0" or Public IP of the machine- Flink
> > crashes
> > > on
> > > > > > start, it doesnt recognize himself.
> > > > > > It is very strange that it doesnt work with 0.0.0.0- basically
> this
> > > is
> > > > a
> > > > > > way in Java to make it being seen widely.
> > > > > > We tried to put there the hostname - what you get from "hostname"
> > > > > command.
> > > > > > It crashes.
> > > > > > It works only with "localhost" and works only locally
> > > > > > So what you suggest we will put there so the remote client could
> > > > connect.
> > > > > > ?
> > > > > >
> > > > > > Best regards
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > > > Do you start Flink via YARN? In that case the
> > > > "jobmanager.rpc.address"
> > > > > is
> > > > > > > not used, because YARN assigns containers/nodes.
> > > > > > >
> > > > > > > If you start Flink in "standalone" mode, this should be the
> > address
> > > > of
> > > > > > the
> > > > > > > node that runs the JobManager. It will be used as the host/IP
> > that
> > > > > Flink
> > > > > > > binds to. The same host should then be used in the
> > > RemoteEnvironment.
> > > > > > >
> > > > > > > Stephan
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello all.
> > > > > > > >
> > > > > > > > Firstly- thank you for your valuable advices.
> > > > > > > > We did some very fine tuned pinpoint test and comes to
> > following
> > > > > > > > conclusions
> > > > > > > >
> > > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > > > 2.Once we copy our Java client program directy to the machine
> > and
> > > > run
> > > > > > it
> > > > > > > > directly there it worked very good
> > > > > > > > The program is
> > > > > > > >
> > > > > > > > .....
> > > > > > > >
> > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("localhost",
> > 6123,
> > > > > > > > "\usr\local\HananTestProj.jar");
> > > > > > > >
> > > > > > > >
> > > > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > > > (org.apache.flink.api.java.DataSet) envRemote.fromElements(
> > > > > > > > "Who's there?",
> > > > > > > > "I think I hear them. Stand, ho! Who's there?");
> > > > > > > >
> > > > > > > > org.apache.flink.api.java.DataSet<Tuple2<String,
> > Integer>>
> > > > > > > > wordCounts = text
> > > > > > > > .flatMap(new LineSplitter())
> > > > > > > > .groupBy(0)
> > > > > > > > .sum(1);
> > > > > > > >
> > > > > > > > wordCounts.print();
> > > > > > > > }
> > > > > > > >
> > > > > > > > public static class LineSplitter implements
> > > > > FlatMapFunction<String,
> > > > > > > > Tuple2<String, Integer>> {
> > > > > > > > public void flatMap(String line,
> > Collector<Tuple2<String,
> > > > > > > Integer>>
> > > > > > > > out) {
> > > > > > > > for (String word : line.split(" ")) {
> > > > > > > > out.collect(new Tuple2<String, Integer>(word,
> > 1));
> > > > > > > > }
> > > > > > > > }
> > > > > > > > }
> > > > > > > >
> > > > > > > > .....
> > > > > > > > Program works fine
> > > > > > > > 3.Now we are trying to run this program remotely , from
> windows
> > > > > machine
> > > > > > > > when the first row looks differently
> > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4",
> 6123,
> > > > > > > > "C:\\HananTestProj.jar");
> > > > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > > > >
> > > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be reached
> > bla
> > > > bla
> > > > > > bla
> > > > > > > >
> > > > > > > > 5.in flink configuration we found a following line
> > > > > > > jobmanager.rpc.address:
> > > > > > > > localhost
> > > > > > > > Flink cant be started with any other value
> (hostname/ipaddress
> > )
> > > > > except
> > > > > > > the
> > > > > > > > localhost
> > > > > > > >
> > > > > > > >
> > > > > > > > 6.We assume that probably Flink has a critical bug : it cant
> be
> > > > > started
> > > > > > > > from remote machine, only locally. Are we right? Are we
> wrong?
> > > > Should
> > > > > > we
> > > > > > > > fill JIRA ?
> > > > > > > > Maybe we need somehow to configure Flink differently?
> > > > > > > >
> > > > > > > > Please advice
> > > > > > > > Best regards
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > The output of the YARN session should look like this:
> > > > > > > > >
> > > > > > > > > Flink JobManager is now running on
> quickstart.cloudera:39956
> > > > > > > > > JobManager Web Interface:
> > > > > > > > >
> > > > > >
> > > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > > > Number of connected TaskManagers changed to 1. Slots
> > > available: 1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The only thing I can think of is that you are not using
> the
> > > > right
> > > > > > > > > host/port
> > > > > > > > > > for the JobManager.
> > > > > > > > > >
> > > > > > > > > > When you start the YARN session, it should print the host
> > > where
> > > > > the
> > > > > > > > > > JobManager runs. You also need to take the port from
> there,
> > > as
> > > > in
> > > > > > > YARN,
> > > > > > > > > the
> > > > > > > > > > port is usually not 6123. Yarn starts many services on
> one
> > > > > machine,
> > > > > > > so
> > > > > > > > > the
> > > > > > > > > > ports need to be randomized.
> > > > > > > > > >
> > > > > > > > > > It may be worth adding a YARNExecutionEnvironment at some
> > > > point,
> > > > > > > which
> > > > > > > > > > deals with this transparent (starting the YARN cluster,
> > > > > connecting
> > > > > > to
> > > > > > > > the
> > > > > > > > > > JobManager).
> > > > > > > > > >
> > > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello.
> > > > > > > > > > > Let me clarify the situation.
> > > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We
> connected
> > it
> > > > to
> > > > > > HDFS
> > > > > > > > > > 2.7.1.
> > > > > > > > > > > 2. Locally, our program is working: once we run flink
> as
> > > > > > > > > > ./start-local.sh,
> > > > > > > > > > > we are able to connect and run the
> > createRemoteEnvironment
> > > > and
> > > > > > > > Execute
> > > > > > > > > > > methods.
> > > > > > > > > > > 3.Due to our architecture and basic Flink feature we
> want
> > > to
> > > > > > invoke
> > > > > > > > > this
> > > > > > > > > > > functionality REMOTELY , when our Java code is calling
> > the
> > > > > Flink
> > > > > > > > > methods
> > > > > > > > > > > from another server.
> > > > > > > > > > > 4.We tried both
> > > > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is
> > > definitely
> > > > > not
> > > > > > > > right
> > > > > > > > > > > since
> > > > > > > > > > > it should be an IP address) - it crash on the "cant
> reach
> > > > > > > JobManager"
> > > > > > > > > > > error.
> > > > > > > > > > >
> > > > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > > > 1.Somehow we need to configure flink to accept the
> > > > connections
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > remote machine
> > > > > > > > > > > 2.Flink has a critical showstopper bug that
> jeopardizing
> > a
> > > > > whole
> > > > > > > > > decision
> > > > > > > > > > > to use this technology.
> > > > > > > > > > >
> > > > > > > > > > > Please advise us how we should advance.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > in the exception you've posted earlier, you can see
> the
> > > > > > following
> > > > > > > > > root
> > > > > > > > > > > > cause:
> > > > > > > > > > > >
> > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not found
> > for:
> > > > > > > > > > > >
> ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > :6123/),
> > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > >
> > > > > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL
> :6123/"
> > > > > usually
> > > > > > > > looks
> > > > > > > > > > like
> > > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it seems
> > that
> > > > you
> > > > > > are
> > > > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > > > Can you pass the correct hostname when you call
> > > > > > > > ExecutionEnvironment.
> > > > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi
> > > > > > > > > > > > > I'm currently using flink 0.9.0 which by maven
> > support
> > > > > > Hadoop
> > > > > > > 1
> > > > > > > > .
> > > > > > > > > > > > > By using flink-clients-0.7.0-hadoop2-incubating.jar
> > > with
> > > > > > > > > > > executePlan(Plan
> > > > > > > > > > > > > p) method instead, I'm getting the same exception
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hanan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. I have restarted Flink service via
> > > > > stop/start-loval.sh -
> > > > > > > it
> > > > > > > > > have
> > > > > > > > > > > > been
> > > > > > > > > > > > > > restarted successfully ,no errors in log folder
> > > > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > Failed
> > > > > > > > > > to
> > > > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > > > at
> > > > > > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > > > at
> > > > > > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > > > at
> > > > > > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> > > :6123/user/jobmanager
> > > > > not
> > > > > > > > > > > reachable.
> > > > > > > > > > > > > > Please make sure that the JobManager is running
> and
> > > its
> > > > > > port
> > > > > > > is
> > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > > > at
> > > > > > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not
> > found
> > > > for:
> > > > > > > > > > > > > >
> > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > :6123/),
> > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > > > at
> > > > > > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > > > >
> > > > > > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > > > >
> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > > > at
> > > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > > > at
> > > > > > > > >
> > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > > > at
> > > > > > > > >
> > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > > > at
> > > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > > > at
> > > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > > > at
> > > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > > > at
> > > > > > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > > > at
> > > akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > > > at
> > > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > > > at
> > > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > > > at
> > > > > > > > > >
> > > > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > >
> > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger
> <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> I guess you are getting an entire exception
> after
> > > the
> > > > > > > > > > > > "org.apache.flink
> > > > > > > > > > > > > >> .client.program.ProgramInvocationException:
> Failed
> > > to
> > > > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > > > >> Can you post it here to help us understanding
> the
> > > > issue?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey
> > Sapozhnikov
> > > <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Some clarification: locally everything works
> > > great.
> > > > > > > > > > > > > >> > However once we run our Flink on remote linux
> > > > machine
> > > > > > and
> > > > > > > > try
> > > > > > > > > to
> > > > > > > > > > > run
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> > client program from our machine, using create
> > > remote
> > > > > > > > > > environment-
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen
> <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > > If you start the job via the "bin/flink"
> > script,
> > > > > then
> > > > > > > > simply
> > > > > > > > > > use
> > > > > > > > > > > > > >> > >
> > "ExecutionEnvironment.getExecutionEnvironment()"
> > > > > > rather
> > > > > > > > then
> > > > > > > > > > > > > creating
> > > > > > > > > > > > > >> a
> > > > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > That way, hosts and ports are configured
> > > > > > automatically.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert
> > Metzger
> > > <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > >> > >> Which values did you use for
> FLINK_SERVER_URL
> > > and
> > > > > > > > > FLINK_PORT?
> > > > > > > > > > > > > >> > >> Every time you deploy Flink on YARN, the
> host
> > > and
> > > > > > port
> > > > > > > > > > change,
> > > > > > > > > > > > > >> because
> > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > >> > >> JobManager is started on a different YARN
> > > > > container.
> > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan
> Meyer
> > <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit Flink to
> > > Yarn
> > > > > > single
> > > > > > > > > node
> > > > > > > > > > > > > cluster
> > > > > > > > > > > > > >> I'm
> > > > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > >
> > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > > >> Failed to
> > > > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > The Jar copy a file from one location in
> > Hdfs
> > > > to
> > > > > > > > another
> > > > > > > > > > and
> > > > > > > > > > > > > works
> > > > > > > > > > > > > >> > fine
> > > > > > > > > > > > > >> > >> > while executed locally on the single node
> > > Yarn
> > > > > > > cluster
> > > > > > > > -
> > > > > > > > > > > > > >> > >> > bin/flink run -c Test
> ./examples/MyJar.jar
> > > > > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > > > > > >> > >> >
> > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > > > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > >
> > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > > > >> > >> >
> > > > > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Regards*
> > > > > >
> > > > > > *Alexey Sapozhnikov*
> > > > > > CTO& Co-Founder
> > > > > > Scalabillit Inc
> > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > M : +972-52-2363823
> > > > > > E : [hidden email]
> > > > > > W : http://www.scalabill.it
> > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > Revolutionizing Proof-of-Concept
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Regards*
> > > >
> > > > *Alexey Sapozhnikov*
> > > > CTO& Co-Founder
> > > > Scalabillit Inc
> > > > Aba Even 10-C, Herzelia, Israel
> > > > M : +972-52-2363823
> > > > E : [hidden email]
> > > > W : http://www.scalabill.it
> > > > YT - https://youtu.be/9Rj309PTOFA
> > > > Map:http://mapta.gs/Scalabillit
> > > > Revolutionizing Proof-of-Concept
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Regards*
> >
> > *Alexey Sapozhnikov*
> > CTO& Co-Founder
> > Scalabillit Inc
> > Aba Even 10-C, Herzelia, Israel
> > M : +972-52-2363823
> > E : [hidden email]
> > W : http://www.scalabill.it
> > YT - https://youtu.be/9Rj309PTOFA
> > Map:http://mapta.gs/Scalabillit
> > Revolutionizing Proof-of-Concept
> >
>

Stephan Ewen

Re: Apache Flink:ProgramInvocationException on Yarn

What you can do as a temporary workaround is to actually enter the IP
address for "jobmanager.rpc.address" - that circumvents the DNS.

Just saw that Akka 2.4 (released some time in the near future) apparently
introduces an option to listen to all network interfaces.

On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov <[hidden email]>
wrote:

> Fully understand.
> 1.My suggestion is to drop Akka and take something else, since this issue
> is really big
> 2.Not hostname not the endpoint are not working, clarifying the VPC topic
> now.
>
> On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen <[hidden email]> wrote:
>
> > Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes
> > annoying, but I have not found a good way around this.
> > The problem is that the address to bind to an the address used by others
> to
> > send messages to the node is the same. (
> > https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY)
> >
> >
> > All of that aside, we have been using Flink on Cloud Platforms GCE,
> Amazon,
> > and it worked.
> >
> > Can you try to use the internal hostname, or the fully qualified external
> > hostname?
> >
> >
> > On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov <[hidden email]
> >
> > wrote:
> >
> > > Hi.
> > > First off - many thanks for your efforts and prompt help.
> > > We will try to find how to do it with DNS server on VPC.
> > > however, absence of "0.0.0.0" is definitely a huge bug - just think
> about
> > > the current situation : if I dont have a VPC, I cant invoke the Flink
> > > functionality remotely in Amazon.
> > > We will clarify the VPC situation and I will write my impression of
> what
> > we
> > > get.
> > >
> > > Thanks, guys
> > >
> > > On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <[hidden email]>
> > > wrote:
> > >
> > > > From this blog post, it seems that this hostname is not resolvable:
> > > >
> > >
> >
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> > > > Can you easily activate a DNS server in the VPC?
> > > >
> > > > 0.0.0.0 is not supported because of some requirements of the Akka
> > > > framework.
> > > > But you should be able to use the host's IP address.
> > > > Can you try that as well?
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Here is the exception from the moment we tried to put in
> > > > > jobmanager.rpc.address the hostname of the machine which is
> > > ip-172-36-98
> > > > > looks like it doesnt recognize this address.
> > > > > Why it doesnt support "0.0.0.0"????
> > > > >
> > > > > 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > -
> > > > >
> > > > >
> > > >
> > >
> >
> --------------------------------------------------------------------------------
> > > > > 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > - Loading configuration from
> > > /usr/local/test/flink-0.9.0/bin/../conf
> > > > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > - Security is not enabled. Starting non-authenticated
> > JobManager.
> > > > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > - Starting JobManager
> > > > > 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > - Starting JobManager actor system at ip-172-36-98:6123.
> > > > > 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
> > > > > - Failed to run JobManager.
> > > > > java.lang.Exception: Could not create JobManager actor system
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > > > > at
> > > > >
> org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > > > > Caused by: java.net.UnknownHostException: ip-172-36-98: unknown
> error
> > > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native
> Method)
> > > > > at
> > > java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > > > > @
> > > > >
> > 21,1
> > > > > 71%
> > > > >
> > > > >
> > > > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > How is Flink crashing when you start it on the Linux machine in
> > > Amazon?
> > > > > >
> > > > > > Can you post the exception here?
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Stephan.
> > > > > > >
> > > > > > > We run this Linux machine on Amazon, which I predict, most of
> the
> > > > > people
> > > > > > > will do.
> > > > > > > We tried to put "0.0.0.0" or Public IP of the machine- Flink
> > > crashes
> > > > on
> > > > > > > start, it doesnt recognize himself.
> > > > > > > It is very strange that it doesnt work with 0.0.0.0- basically
> > this
> > > > is
> > > > > a
> > > > > > > way in Java to make it being seen widely.
> > > > > > > We tried to put there the hostname - what you get from
> "hostname"
> > > > > > command.
> > > > > > > It crashes.
> > > > > > > It works only with "localhost" and works only locally
> > > > > > > So what you suggest we will put there so the remote client
> could
> > > > > connect.
> > > > > > > ?
> > > > > > >
> > > > > > > Best regards
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <
> [hidden email]>
> > > > > wrote:
> > > > > > >
> > > > > > > > Do you start Flink via YARN? In that case the
> > > > > "jobmanager.rpc.address"
> > > > > > is
> > > > > > > > not used, because YARN assigns containers/nodes.
> > > > > > > >
> > > > > > > > If you start Flink in "standalone" mode, this should be the
> > > address
> > > > > of
> > > > > > > the
> > > > > > > > node that runs the JobManager. It will be used as the host/IP
> > > that
> > > > > > Flink
> > > > > > > > binds to. The same host should then be used in the
> > > > RemoteEnvironment.
> > > > > > > >
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello all.
> > > > > > > > >
> > > > > > > > > Firstly- thank you for your valuable advices.
> > > > > > > > > We did some very fine tuned pinpoint test and comes to
> > > following
> > > > > > > > > conclusions
> > > > > > > > >
> > > > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > > > > 2.Once we copy our Java client program directy to the
> machine
> > > and
> > > > > run
> > > > > > > it
> > > > > > > > > directly there it worked very good
> > > > > > > > > The program is
> > > > > > > > >
> > > > > > > > > .....
> > > > > > > > >
> > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("localhost",
> > > 6123,
> > > > > > > > > "\usr\local\HananTestProj.jar");
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > > > > (org.apache.flink.api.java.DataSet) envRemote.fromElements(
> > > > > > > > > "Who's there?",
> > > > > > > > > "I think I hear them. Stand, ho! Who's there?");
> > > > > > > > >
> > > > > > > > > org.apache.flink.api.java.DataSet<Tuple2<String,
> > > Integer>>
> > > > > > > > > wordCounts = text
> > > > > > > > > .flatMap(new LineSplitter())
> > > > > > > > > .groupBy(0)
> > > > > > > > > .sum(1);
> > > > > > > > >
> > > > > > > > > wordCounts.print();
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > public static class LineSplitter implements
> > > > > > FlatMapFunction<String,
> > > > > > > > > Tuple2<String, Integer>> {
> > > > > > > > > public void flatMap(String line,
> > > Collector<Tuple2<String,
> > > > > > > > Integer>>
> > > > > > > > > out) {
> > > > > > > > > for (String word : line.split(" ")) {
> > > > > > > > > out.collect(new Tuple2<String,
> Integer>(word,
> > > 1));
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > .....
> > > > > > > > > Program works fine
> > > > > > > > > 3.Now we are trying to run this program remotely , from
> > windows
> > > > > > machine
> > > > > > > > > when the first row looks differently
> > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4",
> > 6123,
> > > > > > > > > "C:\\HananTestProj.jar");
> > > > > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > > > > >
> > > > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be
> reached
> > > bla
> > > > > bla
> > > > > > > bla
> > > > > > > > >
> > > > > > > > > 5.in flink configuration we found a following line
> > > > > > > > jobmanager.rpc.address:
> > > > > > > > > localhost
> > > > > > > > > Flink cant be started with any other value
> > (hostname/ipaddress
> > > )
> > > > > > except
> > > > > > > > the
> > > > > > > > > localhost
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 6.We assume that probably Flink has a critical bug : it
> cant
> > be
> > > > > > started
> > > > > > > > > from remote machine, only locally. Are we right? Are we
> > wrong?
> > > > > Should
> > > > > > > we
> > > > > > > > > fill JIRA ?
> > > > > > > > > Maybe we need somehow to configure Flink differently?
> > > > > > > > >
> > > > > > > > > Please advice
> > > > > > > > > Best regards
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The output of the YARN session should look like this:
> > > > > > > > > >
> > > > > > > > > > Flink JobManager is now running on
> > quickstart.cloudera:39956
> > > > > > > > > > JobManager Web Interface:
> > > > > > > > > >
> > > > > > >
> > > >
> http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > > > > Number of connected TaskManagers changed to 1. Slots
> > > > available: 1
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The only thing I can think of is that you are not using
> > the
> > > > > right
> > > > > > > > > > host/port
> > > > > > > > > > > for the JobManager.
> > > > > > > > > > >
> > > > > > > > > > > When you start the YARN session, it should print the
> host
> > > > where
> > > > > > the
> > > > > > > > > > > JobManager runs. You also need to take the port from
> > there,
> > > > as
> > > > > in
> > > > > > > > YARN,
> > > > > > > > > > the
> > > > > > > > > > > port is usually not 6123. Yarn starts many services on
> > one
> > > > > > machine,
> > > > > > > > so
> > > > > > > > > > the
> > > > > > > > > > > ports need to be randomized.
> > > > > > > > > > >
> > > > > > > > > > > It may be worth adding a YARNExecutionEnvironment at
> some
> > > > > point,
> > > > > > > > which
> > > > > > > > > > > deals with this transparent (starting the YARN cluster,
> > > > > > connecting
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > JobManager).
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello.
> > > > > > > > > > > > Let me clarify the situation.
> > > > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We
> > connected
> > > it
> > > > > to
> > > > > > > HDFS
> > > > > > > > > > > 2.7.1.
> > > > > > > > > > > > 2. Locally, our program is working: once we run flink
> > as
> > > > > > > > > > > ./start-local.sh,
> > > > > > > > > > > > we are able to connect and run the
> > > createRemoteEnvironment
> > > > > and
> > > > > > > > > Execute
> > > > > > > > > > > > methods.
> > > > > > > > > > > > 3.Due to our architecture and basic Flink feature we
> > want
> > > > to
> > > > > > > invoke
> > > > > > > > > > this
> > > > > > > > > > > > functionality REMOTELY , when our Java code is
> calling
> > > the
> > > > > > Flink
> > > > > > > > > > methods
> > > > > > > > > > > > from another server.
> > > > > > > > > > > > 4.We tried both
> > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is
> > > > definitely
> > > > > > not
> > > > > > > > > right
> > > > > > > > > > > > since
> > > > > > > > > > > > it should be an IP address) - it crash on the "cant
> > reach
> > > > > > > > JobManager"
> > > > > > > > > > > > error.
> > > > > > > > > > > >
> > > > > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > > > > 1.Somehow we need to configure flink to accept the
> > > > > connections
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > remote machine
> > > > > > > > > > > > 2.Flink has a critical showstopper bug that
> > jeopardizing
> > > a
> > > > > > whole
> > > > > > > > > > decision
> > > > > > > > > > > > to use this technology.
> > > > > > > > > > > >
> > > > > > > > > > > > Please advise us how we should advance.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > in the exception you've posted earlier, you can see
> > the
> > > > > > > following
> > > > > > > > > > root
> > > > > > > > > > > > > cause:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not
> found
> > > for:
> > > > > > > > > > > > >
> > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > :6123/),
> > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > >
> > > > > > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL
> > :6123/"
> > > > > > usually
> > > > > > > > > looks
> > > > > > > > > > > like
> > > > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it
> seems
> > > that
> > > > > you
> > > > > > > are
> > > > > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > > > > Can you pass the correct hostname when you call
> > > > > > > > > ExecutionEnvironment.
> > > > > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > I'm currently using flink 0.9.0 which by maven
> > > support
> > > > > > > Hadoop
> > > > > > > > 1
> > > > > > > > > .
> > > > > > > > > > > > > > By using
> flink-clients-0.7.0-hadoop2-incubating.jar
> > > > with
> > > > > > > > > > > > executePlan(Plan
> > > > > > > > > > > > > > p) method instead, I'm getting the same
> exception
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hanan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. I have restarted Flink service via
> > > > > > stop/start-loval.sh -
> > > > > > > > it
> > > > > > > > > > have
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > restarted successfully ,no errors in log folder
> > > > > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > Failed
> > > > > > > > > > > to
> > > > > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > > > > at
> > > > > > > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > > > > at
> > > > > > > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > > > > at
> > > > > > > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> > > > :6123/user/jobmanager
> > > > > > not
> > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > Please make sure that the JobManager is running
> > and
> > > > its
> > > > > > > port
> > > > > > > > is
> > > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > > > > at
> > > > > > > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not
> > > found
> > > > > for:
> > > > > > > > > > > > > > >
> > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > > :6123/),
> > > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > > > > at
> > > > > > > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > > > > >
> > > > > > > >
> Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > >
> > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > > > > at
> > > > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > > > > at
> > > > > > > > > >
> > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > > > > at
> > > > > > > > > >
> > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > > > > at
> > > > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > > > > at
> > > > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > > > > at
> > > > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > > > > at
> > > > > > > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > > > > at
> > > > akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > > > > at
> > > > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > > > > at
> > > > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > > > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > > > > at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > > > > at
> akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > >
> > > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert
> Metzger
> > <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> I guess you are getting an entire exception
> > after
> > > > the
> > > > > > > > > > > > > "org.apache.flink
> > > > > > > > > > > > > > >> .client.program.ProgramInvocationException:
> > Failed
> > > > to
> > > > > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > > > > >> Can you post it here to help us understanding
> > the
> > > > > issue?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey
> > > Sapozhnikov
> > > > <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Some clarification: locally everything works
> > > > great.
> > > > > > > > > > > > > > >> > However once we run our Flink on remote
> linux
> > > > > machine
> > > > > > > and
> > > > > > > > > try
> > > > > > > > > > to
> > > > > > > > > > > > run
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> > client program from our machine, using
> create
> > > > remote
> > > > > > > > > > > environment-
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan
> Ewen
> > <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > > If you start the job via the "bin/flink"
> > > script,
> > > > > > then
> > > > > > > > > simply
> > > > > > > > > > > use
> > > > > > > > > > > > > > >> > >
> > > "ExecutionEnvironment.getExecutionEnvironment()"
> > > > > > > rather
> > > > > > > > > then
> > > > > > > > > > > > > > creating
> > > > > > > > > > > > > > >> a
> > > > > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > That way, hosts and ports are configured
> > > > > > > automatically.
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert
> > > Metzger
> > > > <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> Which values did you use for
> > FLINK_SERVER_URL
> > > > and
> > > > > > > > > > FLINK_PORT?
> > > > > > > > > > > > > > >> > >> Every time you deploy Flink on YARN, the
> > host
> > > > and
> > > > > > > port
> > > > > > > > > > > change,
> > > > > > > > > > > > > > >> because
> > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > >> > >> JobManager is started on a different YARN
> > > > > > container.
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan
> > Meyer
> > > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit Flink
> to
> > > > Yarn
> > > > > > > single
> > > > > > > > > > node
> > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > >> I'm
> > > > > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > >
> > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > > > >> Failed to
> > > > > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > The Jar copy a file from one location
> in
> > > Hdfs
> > > > > to
> > > > > > > > > another
> > > > > > > > > > > and
> > > > > > > > > > > > > > works
> > > > > > > > > > > > > > >> > fine
> > > > > > > > > > > > > > >> > >> > while executed locally on the single
> node
> > > > Yarn
> > > > > > > > cluster
> > > > > > > > > -
> > > > > > > > > > > > > > >> > >> > bin/flink run -c Test
> > ./examples/MyJar.jar
> > > > > > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > > > > > > >> > >> >
> > > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > > > > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > >
> > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *Regards*
> > > > > > >
> > > > > > > *Alexey Sapozhnikov*
> > > > > > > CTO& Co-Founder
> > > > > > > Scalabillit Inc
> > > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > > M : +972-52-2363823
> > > > > > > E : [hidden email]
> > > > > > > W : http://www.scalabill.it
> > > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > > Revolutionizing Proof-of-Concept
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Regards*
> > > > >
> > > > > *Alexey Sapozhnikov*
> > > > > CTO& Co-Founder
> > > > > Scalabillit Inc
> > > > > Aba Even 10-C, Herzelia, Israel
> > > > > M : +972-52-2363823
> > > > > E : [hidden email]
> > > > > W : http://www.scalabill.it
> > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > Map:http://mapta.gs/Scalabillit
> > > > > Revolutionizing Proof-of-Concept
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Regards*
> > >
> > > *Alexey Sapozhnikov*
> > > CTO& Co-Founder
> > > Scalabillit Inc
> > > Aba Even 10-C, Herzelia, Israel
> > > M : +972-52-2363823
> > > E : [hidden email]
> > > W : http://www.scalabill.it
> > > YT - https://youtu.be/9Rj309PTOFA
> > > Map:http://mapta.gs/Scalabillit
> > > Revolutionizing Proof-of-Concept
> > >
> >
>
>
>
> --
>
> *Regards*
>
> *Alexey Sapozhnikov*
> CTO& Co-Founder
> Scalabillit Inc
> Aba Even 10-C, Herzelia, Israel
> M : +972-52-2363823
> E : [hidden email]
> W : http://www.scalabill.it
> YT - https://youtu.be/9Rj309PTOFA
> Map:http://mapta.gs/Scalabillit
> Revolutionizing Proof-of-Concept
>

Alexey Sapozhnikov

Re: Apache Flink:ProgramInvocationException on Yarn

it will not help, since the internal IPs are changing in AWS from time to
time and you should use only Public IP, which is not recognizable by flink.
Thats why all app servers, for example JBoss or even Flume are using
"0.0.0.0"

On Sun, Aug 30, 2015 at 5:53 PM, Stephan Ewen <[hidden email]> wrote:

> What you can do as a temporary workaround is to actually enter the IP
> address for "jobmanager.rpc.address" - that circumvents the DNS.
>
> Just saw that Akka 2.4 (released some time in the near future) apparently
> introduces an option to listen to all network interfaces.
>
> On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov <[hidden email]>
> wrote:
>
> > Fully understand.
> > 1.My suggestion is to drop Akka and take something else, since this issue
> > is really big
> > 2.Not hostname not the endpoint are not working, clarifying the VPC topic
> > now.
> >
> > On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes
> > > annoying, but I have not found a good way around this.
> > > The problem is that the address to bind to an the address used by
> others
> > to
> > > send messages to the node is the same. (
> > > https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY)
> > >
> > >
> > > All of that aside, we have been using Flink on Cloud Platforms GCE,
> > Amazon,
> > > and it worked.
> > >
> > > Can you try to use the internal hostname, or the fully qualified
> external
> > > hostname?
> > >
> > >
> > > On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Hi.
> > > > First off - many thanks for your efforts and prompt help.
> > > > We will try to find how to do it with DNS server on VPC.
> > > > however, absence of "0.0.0.0" is definitely a huge bug - just think
> > about
> > > > the current situation : if I dont have a VPC, I cant invoke the Flink
> > > > functionality remotely in Amazon.
> > > > We will clarify the VPC situation and I will write my impression of
> > what
> > > we
> > > > get.
> > > >
> > > > Thanks, guys
> > > >
> > > > On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > From this blog post, it seems that this hostname is not resolvable:
> > > > >
> > > >
> > >
> >
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> > > > > Can you easily activate a DNS server in the VPC?
> > > > >
> > > > > 0.0.0.0 is not supported because of some requirements of the Akka
> > > > > framework.
> > > > > But you should be able to use the host's IP address.
> > > > > Can you try that as well?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Here is the exception from the moment we tried to put in
> > > > > > jobmanager.rpc.address the hostname of the machine which is
> > > > ip-172-36-98
> > > > > > looks like it doesnt recognize this address.
> > > > > > Why it doesnt support "0.0.0.0"????
> > > > > >
> > > > > > 13:43:14,805 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > > -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> --------------------------------------------------------------------------------
> > > > > > 13:43:14,914 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > > - Loading configuration from
> > > > /usr/local/test/flink-0.9.0/bin/../conf
> > > > > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > > - Security is not enabled. Starting non-authenticated
> > > JobManager.
> > > > > > 13:43:14,942 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > > - Starting JobManager
> > > > > > 13:43:14,944 INFO org.apache.flink.runtime.jobmanager.JobManager
> > > > > > - Starting JobManager actor system at ip-172-36-98:6123.
> > > > > > 13:43:15,047 ERROR org.apache.flink.runtime.jobmanager.JobManager
> > > > > > - Failed to run JobManager.
> > > > > > java.lang.Exception: Could not create JobManager actor system
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > > > > > at
> > > > > >
> > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > > > > > Caused by: java.net.UnknownHostException: ip-172-36-98: unknown
> > error
> > > > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native
> > Method)
> > > > > > at
> > > > java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > > > > > @
> > > > > >
> > > 21,1
> > > > > > 71%
> > > > > >
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > How is Flink crashing when you start it on the Linux machine in
> > > > Amazon?
> > > > > > >
> > > > > > > Can you post the exception here?
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Stephan.
> > > > > > > >
> > > > > > > > We run this Linux machine on Amazon, which I predict, most of
> > the
> > > > > > people
> > > > > > > > will do.
> > > > > > > > We tried to put "0.0.0.0" or Public IP of the machine- Flink
> > > > crashes
> > > > > on
> > > > > > > > start, it doesnt recognize himself.
> > > > > > > > It is very strange that it doesnt work with 0.0.0.0-
> basically
> > > this
> > > > > is
> > > > > > a
> > > > > > > > way in Java to make it being seen widely.
> > > > > > > > We tried to put there the hostname - what you get from
> > "hostname"
> > > > > > > command.
> > > > > > > > It crashes.
> > > > > > > > It works only with "localhost" and works only locally
> > > > > > > > So what you suggest we will put there so the remote client
> > could
> > > > > > connect.
> > > > > > > > ?
> > > > > > > >
> > > > > > > > Best regards
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Do you start Flink via YARN? In that case the
> > > > > > "jobmanager.rpc.address"
> > > > > > > is
> > > > > > > > > not used, because YARN assigns containers/nodes.
> > > > > > > > >
> > > > > > > > > If you start Flink in "standalone" mode, this should be the
> > > > address
> > > > > > of
> > > > > > > > the
> > > > > > > > > node that runs the JobManager. It will be used as the
> host/IP
> > > > that
> > > > > > > Flink
> > > > > > > > > binds to. The same host should then be used in the
> > > > > RemoteEnvironment.
> > > > > > > > >
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello all.
> > > > > > > > > >
> > > > > > > > > > Firstly- thank you for your valuable advices.
> > > > > > > > > > We did some very fine tuned pinpoint test and comes to
> > > > following
> > > > > > > > > > conclusions
> > > > > > > > > >
> > > > > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > > > > > 2.Once we copy our Java client program directy to the
> > machine
> > > > and
> > > > > > run
> > > > > > > > it
> > > > > > > > > > directly there it worked very good
> > > > > > > > > > The program is
> > > > > > > > > >
> > > > > > > > > > .....
> > > > > > > > > >
> > > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > >
> =ExecutionEnvironment.createRemoteEnvironment("localhost",
> > > > 6123,
> > > > > > > > > > "\usr\local\HananTestProj.jar");
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > > > > > (org.apache.flink.api.java.DataSet)
> envRemote.fromElements(
> > > > > > > > > > "Who's there?",
> > > > > > > > > > "I think I hear them. Stand, ho! Who's
> there?");
> > > > > > > > > >
> > > > > > > > > > org.apache.flink.api.java.DataSet<Tuple2<String,
> > > > Integer>>
> > > > > > > > > > wordCounts = text
> > > > > > > > > > .flatMap(new LineSplitter())
> > > > > > > > > > .groupBy(0)
> > > > > > > > > > .sum(1);
> > > > > > > > > >
> > > > > > > > > > wordCounts.print();
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > public static class LineSplitter implements
> > > > > > > FlatMapFunction<String,
> > > > > > > > > > Tuple2<String, Integer>> {
> > > > > > > > > > public void flatMap(String line,
> > > > Collector<Tuple2<String,
> > > > > > > > > Integer>>
> > > > > > > > > > out) {
> > > > > > > > > > for (String word : line.split(" ")) {
> > > > > > > > > > out.collect(new Tuple2<String,
> > Integer>(word,
> > > > 1));
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > .....
> > > > > > > > > > Program works fine
> > > > > > > > > > 3.Now we are trying to run this program remotely , from
> > > windows
> > > > > > > machine
> > > > > > > > > > when the first row looks differently
> > > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4",
> > > 6123,
> > > > > > > > > > "C:\\HananTestProj.jar");
> > > > > > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > > > > > >
> > > > > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be
> > reached
> > > > bla
> > > > > > bla
> > > > > > > > bla
> > > > > > > > > >
> > > > > > > > > > 5.in flink configuration we found a following line
> > > > > > > > > jobmanager.rpc.address:
> > > > > > > > > > localhost
> > > > > > > > > > Flink cant be started with any other value
> > > (hostname/ipaddress
> > > > )
> > > > > > > except
> > > > > > > > > the
> > > > > > > > > > localhost
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 6.We assume that probably Flink has a critical bug : it
> > cant
> > > be
> > > > > > > started
> > > > > > > > > > from remote machine, only locally. Are we right? Are we
> > > wrong?
> > > > > > Should
> > > > > > > > we
> > > > > > > > > > fill JIRA ?
> > > > > > > > > > Maybe we need somehow to configure Flink differently?
> > > > > > > > > >
> > > > > > > > > > Please advice
> > > > > > > > > > Best regards
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The output of the YARN session should look like this:
> > > > > > > > > > >
> > > > > > > > > > > Flink JobManager is now running on
> > > quickstart.cloudera:39956
> > > > > > > > > > > JobManager Web Interface:
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > > > > > Number of connected TaskManagers changed to 1. Slots
> > > > > available: 1
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The only thing I can think of is that you are not
> using
> > > the
> > > > > > right
> > > > > > > > > > > host/port
> > > > > > > > > > > > for the JobManager.
> > > > > > > > > > > >
> > > > > > > > > > > > When you start the YARN session, it should print the
> > host
> > > > > where
> > > > > > > the
> > > > > > > > > > > > JobManager runs. You also need to take the port from
> > > there,
> > > > > as
> > > > > > in
> > > > > > > > > YARN,
> > > > > > > > > > > the
> > > > > > > > > > > > port is usually not 6123. Yarn starts many services
> on
> > > one
> > > > > > > machine,
> > > > > > > > > so
> > > > > > > > > > > the
> > > > > > > > > > > > ports need to be randomized.
> > > > > > > > > > > >
> > > > > > > > > > > > It may be worth adding a YARNExecutionEnvironment at
> > some
> > > > > > point,
> > > > > > > > > which
> > > > > > > > > > > > deals with this transparent (starting the YARN
> cluster,
> > > > > > > connecting
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > JobManager).
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hello.
> > > > > > > > > > > > > Let me clarify the situation.
> > > > > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We
> > > connected
> > > > it
> > > > > > to
> > > > > > > > HDFS
> > > > > > > > > > > > 2.7.1.
> > > > > > > > > > > > > 2. Locally, our program is working: once we run
> flink
> > > as
> > > > > > > > > > > > ./start-local.sh,
> > > > > > > > > > > > > we are able to connect and run the
> > > > createRemoteEnvironment
> > > > > > and
> > > > > > > > > > Execute
> > > > > > > > > > > > > methods.
> > > > > > > > > > > > > 3.Due to our architecture and basic Flink feature
> we
> > > want
> > > > > to
> > > > > > > > invoke
> > > > > > > > > > > this
> > > > > > > > > > > > > functionality REMOTELY , when our Java code is
> > calling
> > > > the
> > > > > > > Flink
> > > > > > > > > > > methods
> > > > > > > > > > > > > from another server.
> > > > > > > > > > > > > 4.We tried both
> > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is
> > > > > definitely
> > > > > > > not
> > > > > > > > > > right
> > > > > > > > > > > > > since
> > > > > > > > > > > > > it should be an IP address) - it crash on the "cant
> > > reach
> > > > > > > > > JobManager"
> > > > > > > > > > > > > error.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > > > > > 1.Somehow we need to configure flink to accept the
> > > > > > connections
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > remote machine
> > > > > > > > > > > > > 2.Flink has a critical showstopper bug that
> > > jeopardizing
> > > > a
> > > > > > > whole
> > > > > > > > > > > decision
> > > > > > > > > > > > > to use this technology.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please advise us how we should advance.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > in the exception you've posted earlier, you can
> see
> > > the
> > > > > > > > following
> > > > > > > > > > > root
> > > > > > > > > > > > > > cause:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not
> > found
> > > > for:
> > > > > > > > > > > > > >
> > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > :6123/),
> > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL
> > > :6123/"
> > > > > > > usually
> > > > > > > > > > looks
> > > > > > > > > > > > like
> > > > > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it
> > seems
> > > > that
> > > > > > you
> > > > > > > > are
> > > > > > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > > > > > Can you pass the correct hostname when you call
> > > > > > > > > > ExecutionEnvironment.
> > > > > > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > I'm currently using flink 0.9.0 which by maven
> > > > support
> > > > > > > > Hadoop
> > > > > > > > > 1
> > > > > > > > > > .
> > > > > > > > > > > > > > > By using
> > flink-clients-0.7.0-hadoop2-incubating.jar
> > > > > with
> > > > > > > > > > > > > executePlan(Plan
> > > > > > > > > > > > > > > p) method instead, I'm getting the same
> > exception
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hanan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. I have restarted Flink service via
> > > > > > > stop/start-loval.sh -
> > > > > > > > > it
> > > > > > > > > > > have
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > restarted successfully ,no errors in log
> folder
> > > > > > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > Failed
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > > > > > at
> > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > > > > > at
> > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > > > > > at
> > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > > > > > Caused by: java.io.IOException: JobManager at
> > > > > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> > > > > :6123/user/jobmanager
> > > > > > > not
> > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > > Please make sure that the JobManager is
> running
> > > and
> > > > > its
> > > > > > > > port
> > > > > > > > > is
> > > > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > > > > > at
> > > > > > > > > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor
> not
> > > > found
> > > > > > for:
> > > > > > > > > > > > > > > >
> > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > > > :6123/),
> > > > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > > > > > at
> > > > > > > > >
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > >
> > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > > > > > at
> > > > > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > > > > > at
> > > > > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > > > > > at
> > > > > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > > > > > at
> > > > > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > > > > > at
> > > > > > > > >
> akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > > > > > at
> > > > > akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > > > > > at
> > > > > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > > > > > at
> > > > > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > > > > > at
> akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > > > > > at
> > akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert
> > Metzger
> > > <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> I guess you are getting an entire exception
> > > after
> > > > > the
> > > > > > > > > > > > > > "org.apache.flink
> > > > > > > > > > > > > > > >> .client.program.ProgramInvocationException:
> > > Failed
> > > > > to
> > > > > > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > > > > > >> Can you post it here to help us
> understanding
> > > the
> > > > > > issue?
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey
> > > > Sapozhnikov
> > > > > <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Some clarification: locally everything
> works
> > > > > great.
> > > > > > > > > > > > > > > >> > However once we run our Flink on remote
> > linux
> > > > > > machine
> > > > > > > > and
> > > > > > > > > > try
> > > > > > > > > > > to
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> > client program from our machine, using
> > create
> > > > > remote
> > > > > > > > > > > > environment-
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan
> > Ewen
> > > <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > > If you start the job via the "bin/flink"
> > > > script,
> > > > > > > then
> > > > > > > > > > simply
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > >> > >
> > > > "ExecutionEnvironment.getExecutionEnvironment()"
> > > > > > > > rather
> > > > > > > > > > then
> > > > > > > > > > > > > > > creating
> > > > > > > > > > > > > > > >> a
> > > > > > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > That way, hosts and ports are configured
> > > > > > > > automatically.
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM, Robert
> > > > Metzger
> > > > > <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > >> > >> Which values did you use for
> > > FLINK_SERVER_URL
> > > > > and
> > > > > > > > > > > FLINK_PORT?
> > > > > > > > > > > > > > > >> > >> Every time you deploy Flink on YARN,
> the
> > > host
> > > > > and
> > > > > > > > port
> > > > > > > > > > > > change,
> > > > > > > > > > > > > > > >> because
> > > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > > >> > >> JobManager is started on a different
> YARN
> > > > > > > container.
> > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan
> > > Meyer
> > > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit
> Flink
> > to
> > > > > Yarn
> > > > > > > > single
> > > > > > > > > > > node
> > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > >> I'm
> > > > > > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > >
> > > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > > > > >> Failed to
> > > > > > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > The Jar copy a file from one location
> > in
> > > > Hdfs
> > > > > > to
> > > > > > > > > > another
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > >> > fine
> > > > > > > > > > > > > > > >> > >> > while executed locally on the single
> > node
> > > > > Yarn
> > > > > > > > > cluster
> > > > > > > > > > -
> > > > > > > > > > > > > > > >> > >> > bin/flink run -c Test
> > > ./examples/MyJar.jar
> > > > > > > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > ExecutionEnvironment envRemote =
> > > > > > > > > > > > > > > >> > >> >
> > > > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > > > > > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > >
> > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > *Regards*
> > > > > > > >
> > > > > > > > *Alexey Sapozhnikov*
> > > > > > > > CTO& Co-Founder
> > > > > > > > Scalabillit Inc
> > > > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > > > M : +972-52-2363823
> > > > > > > > E : [hidden email]
> > > > > > > > W : http://www.scalabill.it
> > > > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > > > Revolutionizing Proof-of-Concept
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Regards*
> > > > > >
> > > > > > *Alexey Sapozhnikov*
> > > > > > CTO& Co-Founder
> > > > > > Scalabillit Inc
> > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > M : +972-52-2363823
> > > > > > E : [hidden email]
> > > > > > W : http://www.scalabill.it
> > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > Revolutionizing Proof-of-Concept
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Regards*
> > > >
> > > > *Alexey Sapozhnikov*
> > > > CTO& Co-Founder
> > > > Scalabillit Inc
> > > > Aba Even 10-C, Herzelia, Israel
> > > > M : +972-52-2363823
> > > > E : [hidden email]
> > > > W : http://www.scalabill.it
> > > > YT - https://youtu.be/9Rj309PTOFA
> > > > Map:http://mapta.gs/Scalabillit
> > > > Revolutionizing Proof-of-Concept
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Regards*
> >
> > *Alexey Sapozhnikov*
> > CTO& Co-Founder
> > Scalabillit Inc
> > Aba Even 10-C, Herzelia, Israel
> > M : +972-52-2363823
> > E : [hidden email]
> > W : http://www.scalabill.it
> > YT - https://youtu.be/9Rj309PTOFA
> > Map:http://mapta.gs/Scalabillit
> > Revolutionizing Proof-of-Concept
> >
>

Stephan Ewen

Re: Apache Flink:ProgramInvocationException on Yarn

Why are the external IPs not working? Any kind of exception you can share?

On Sun, Aug 30, 2015 at 5:02 PM, Alexey Sapozhnikov <[hidden email]>
wrote:

> it will not help, since the internal IPs are changing in AWS from time to
> time and you should use only Public IP, which is not recognizable by flink.
> Thats why all app servers, for example JBoss or even Flume are using
> "0.0.0.0"
>
> On Sun, Aug 30, 2015 at 5:53 PM, Stephan Ewen <[hidden email]> wrote:
>
> > What you can do as a temporary workaround is to actually enter the IP
> > address for "jobmanager.rpc.address" - that circumvents the DNS.
> >
> > Just saw that Akka 2.4 (released some time in the near future) apparently
> > introduces an option to listen to all network interfaces.
> >
> > On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov <[hidden email]
> >
> > wrote:
> >
> > > Fully understand.
> > > 1.My suggestion is to drop Akka and take something else, since this
> issue
> > > is really big
> > > 2.Not hostname not the endpoint are not working, clarifying the VPC
> topic
> > > now.
> > >
> > > On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen <[hidden email]>
> wrote:
> > >
> > > > Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes
> > > > annoying, but I have not found a good way around this.
> > > > The problem is that the address to bind to an the address used by
> > others
> > > to
> > > > send messages to the node is the same. (
> > > > https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY)
> > > >
> > > >
> > > > All of that aside, we have been using Flink on Cloud Platforms GCE,
> > > Amazon,
> > > > and it worked.
> > > >
> > > > Can you try to use the internal hostname, or the fully qualified
> > external
> > > > hostname?
> > > >
> > > >
> > > > On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov <
> > [hidden email]
> > > >
> > > > wrote:
> > > >
> > > > > Hi.
> > > > > First off - many thanks for your efforts and prompt help.
> > > > > We will try to find how to do it with DNS server on VPC.
> > > > > however, absence of "0.0.0.0" is definitely a huge bug - just think
> > > about
> > > > > the current situation : if I dont have a VPC, I cant invoke the
> Flink
> > > > > functionality remotely in Amazon.
> > > > > We will clarify the VPC situation and I will write my impression of
> > > what
> > > > we
> > > > > get.
> > > > >
> > > > > Thanks, guys
> > > > >
> > > > > On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > From this blog post, it seems that this hostname is not
> resolvable:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> > > > > > Can you easily activate a DNS server in the VPC?
> > > > > >
> > > > > > 0.0.0.0 is not supported because of some requirements of the Akka
> > > > > > framework.
> > > > > > But you should be able to use the host's IP address.
> > > > > > Can you try that as well?
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <
> > > > [hidden email]
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Here is the exception from the moment we tried to put in
> > > > > > > jobmanager.rpc.address the hostname of the machine which is
> > > > > ip-172-36-98
> > > > > > > looks like it doesnt recognize this address.
> > > > > > > Why it doesnt support "0.0.0.0"????
> > > > > > >
> > > > > > > 13:43:14,805 INFO
> org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > -
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> --------------------------------------------------------------------------------
> > > > > > > 13:43:14,914 INFO
> org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > - Loading configuration from
> > > > > /usr/local/test/flink-0.9.0/bin/../conf
> > > > > > > 13:43:14,942 INFO
> org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > - Security is not enabled. Starting non-authenticated
> > > > JobManager.
> > > > > > > 13:43:14,942 INFO
> org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > - Starting JobManager
> > > > > > > 13:43:14,944 INFO
> org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > - Starting JobManager actor system at ip-172-36-98:6123.
> > > > > > > 13:43:15,047 ERROR
> org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > - Failed to run JobManager.
> > > > > > > java.lang.Exception: Could not create JobManager actor system
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > > > > > > at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > > > > > > at
> > > > > > >
> > > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > > > > > > Caused by: java.net.UnknownHostException: ip-172-36-98: unknown
> > > error
> > > > > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native
> > > Method)
> > > > > > > at
> > > > > java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > > > > > > @
> > > > > > >
> > > > 21,1
> > > > > > > 71%
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > How is Flink crashing when you start it on the Linux machine
> in
> > > > > Amazon?
> > > > > > > >
> > > > > > > > Can you post the exception here?
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Stephan.
> > > > > > > > >
> > > > > > > > > We run this Linux machine on Amazon, which I predict, most
> of
> > > the
> > > > > > > people
> > > > > > > > > will do.
> > > > > > > > > We tried to put "0.0.0.0" or Public IP of the machine-
> Flink
> > > > > crashes
> > > > > > on
> > > > > > > > > start, it doesnt recognize himself.
> > > > > > > > > It is very strange that it doesnt work with 0.0.0.0-
> > basically
> > > > this
> > > > > > is
> > > > > > > a
> > > > > > > > > way in Java to make it being seen widely.
> > > > > > > > > We tried to put there the hostname - what you get from
> > > "hostname"
> > > > > > > > command.
> > > > > > > > > It crashes.
> > > > > > > > > It works only with "localhost" and works only locally
> > > > > > > > > So what you suggest we will put there so the remote client
> > > could
> > > > > > > connect.
> > > > > > > > > ?
> > > > > > > > >
> > > > > > > > > Best regards
> > > > > > > > >
> > > > > > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do you start Flink via YARN? In that case the
> > > > > > > "jobmanager.rpc.address"
> > > > > > > > is
> > > > > > > > > > not used, because YARN assigns containers/nodes.
> > > > > > > > > >
> > > > > > > > > > If you start Flink in "standalone" mode, this should be
> the
> > > > > address
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > node that runs the JobManager. It will be used as the
> > host/IP
> > > > > that
> > > > > > > > Flink
> > > > > > > > > > binds to. The same host should then be used in the
> > > > > > RemoteEnvironment.
> > > > > > > > > >
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello all.
> > > > > > > > > > >
> > > > > > > > > > > Firstly- thank you for your valuable advices.
> > > > > > > > > > > We did some very fine tuned pinpoint test and comes to
> > > > > following
> > > > > > > > > > > conclusions
> > > > > > > > > > >
> > > > > > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > > > > > > 2.Once we copy our Java client program directy to the
> > > machine
> > > > > and
> > > > > > > run
> > > > > > > > > it
> > > > > > > > > > > directly there it worked very good
> > > > > > > > > > > The program is
> > > > > > > > > > >
> > > > > > > > > > > .....
> > > > > > > > > > >
> > > > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > > >
> > =ExecutionEnvironment.createRemoteEnvironment("localhost",
> > > > > 6123,
> > > > > > > > > > > "\usr\local\HananTestProj.jar");
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > > > > > > (org.apache.flink.api.java.DataSet)
> > envRemote.fromElements(
> > > > > > > > > > > "Who's there?",
> > > > > > > > > > > "I think I hear them. Stand, ho! Who's
> > there?");
> > > > > > > > > > >
> > > > > > > > > > > org.apache.flink.api.java.DataSet<Tuple2<String,
> > > > > Integer>>
> > > > > > > > > > > wordCounts = text
> > > > > > > > > > > .flatMap(new LineSplitter())
> > > > > > > > > > > .groupBy(0)
> > > > > > > > > > > .sum(1);
> > > > > > > > > > >
> > > > > > > > > > > wordCounts.print();
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > public static class LineSplitter implements
> > > > > > > > FlatMapFunction<String,
> > > > > > > > > > > Tuple2<String, Integer>> {
> > > > > > > > > > > public void flatMap(String line,
> > > > > Collector<Tuple2<String,
> > > > > > > > > > Integer>>
> > > > > > > > > > > out) {
> > > > > > > > > > > for (String word : line.split(" ")) {
> > > > > > > > > > > out.collect(new Tuple2<String,
> > > Integer>(word,
> > > > > 1));
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > .....
> > > > > > > > > > > Program works fine
> > > > > > > > > > > 3.Now we are trying to run this program remotely , from
> > > > windows
> > > > > > > > machine
> > > > > > > > > > > when the first row looks differently
> > > > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > > >
> =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4",
> > > > 6123,
> > > > > > > > > > > "C:\\HananTestProj.jar");
> > > > > > > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > > > > > > >
> > > > > > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be
> > > reached
> > > > > bla
> > > > > > > bla
> > > > > > > > > bla
> > > > > > > > > > >
> > > > > > > > > > > 5.in flink configuration we found a following line
> > > > > > > > > > jobmanager.rpc.address:
> > > > > > > > > > > localhost
> > > > > > > > > > > Flink cant be started with any other value
> > > > (hostname/ipaddress
> > > > > )
> > > > > > > > except
> > > > > > > > > > the
> > > > > > > > > > > localhost
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 6.We assume that probably Flink has a critical bug : it
> > > cant
> > > > be
> > > > > > > > started
> > > > > > > > > > > from remote machine, only locally. Are we right? Are we
> > > > wrong?
> > > > > > > Should
> > > > > > > > > we
> > > > > > > > > > > fill JIRA ?
> > > > > > > > > > > Maybe we need somehow to configure Flink differently?
> > > > > > > > > > >
> > > > > > > > > > > Please advice
> > > > > > > > > > > Best regards
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The output of the YARN session should look like this:
> > > > > > > > > > > >
> > > > > > > > > > > > Flink JobManager is now running on
> > > > quickstart.cloudera:39956
> > > > > > > > > > > > JobManager Web Interface:
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > > > > > > Number of connected TaskManagers changed to 1. Slots
> > > > > > available: 1
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The only thing I can think of is that you are not
> > using
> > > > the
> > > > > > > right
> > > > > > > > > > > > host/port
> > > > > > > > > > > > > for the JobManager.
> > > > > > > > > > > > >
> > > > > > > > > > > > > When you start the YARN session, it should print
> the
> > > host
> > > > > > where
> > > > > > > > the
> > > > > > > > > > > > > JobManager runs. You also need to take the port
> from
> > > > there,
> > > > > > as
> > > > > > > in
> > > > > > > > > > YARN,
> > > > > > > > > > > > the
> > > > > > > > > > > > > port is usually not 6123. Yarn starts many services
> > on
> > > > one
> > > > > > > > machine,
> > > > > > > > > > so
> > > > > > > > > > > > the
> > > > > > > > > > > > > ports need to be randomized.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It may be worth adding a YARNExecutionEnvironment
> at
> > > some
> > > > > > > point,
> > > > > > > > > > which
> > > > > > > > > > > > > deals with this transparent (starting the YARN
> > cluster,
> > > > > > > > connecting
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > JobManager).
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello.
> > > > > > > > > > > > > > Let me clarify the situation.
> > > > > > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We
> > > > connected
> > > > > it
> > > > > > > to
> > > > > > > > > HDFS
> > > > > > > > > > > > > 2.7.1.
> > > > > > > > > > > > > > 2. Locally, our program is working: once we run
> > flink
> > > > as
> > > > > > > > > > > > > ./start-local.sh,
> > > > > > > > > > > > > > we are able to connect and run the
> > > > > createRemoteEnvironment
> > > > > > > and
> > > > > > > > > > > Execute
> > > > > > > > > > > > > > methods.
> > > > > > > > > > > > > > 3.Due to our architecture and basic Flink feature
> > we
> > > > want
> > > > > > to
> > > > > > > > > invoke
> > > > > > > > > > > > this
> > > > > > > > > > > > > > functionality REMOTELY , when our Java code is
> > > calling
> > > > > the
> > > > > > > > Flink
> > > > > > > > > > > > methods
> > > > > > > > > > > > > > from another server.
> > > > > > > > > > > > > > 4.We tried both
> > > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which is
> > > > > > definitely
> > > > > > > > not
> > > > > > > > > > > right
> > > > > > > > > > > > > > since
> > > > > > > > > > > > > > it should be an IP address) - it crash on the
> "cant
> > > > reach
> > > > > > > > > > JobManager"
> > > > > > > > > > > > > > error.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > > > > > > 1.Somehow we need to configure flink to accept
> the
> > > > > > > connections
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > remote machine
> > > > > > > > > > > > > > 2.Flink has a critical showstopper bug that
> > > > jeopardizing
> > > > > a
> > > > > > > > whole
> > > > > > > > > > > > decision
> > > > > > > > > > > > > > to use this technology.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please advise us how we should advance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert Metzger
> <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > in the exception you've posted earlier, you can
> > see
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > root
> > > > > > > > > > > > > > > cause:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor not
> > > found
> > > > > for:
> > > > > > > > > > > > > > >
> > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > > :6123/),
> > > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This string "akka.tcp://flink@FLINK_SERVER_URL
> > > > :6123/"
> > > > > > > > usually
> > > > > > > > > > > looks
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So it
> > > seems
> > > > > that
> > > > > > > you
> > > > > > > > > are
> > > > > > > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > > > > > > Can you pass the correct hostname when you call
> > > > > > > > > > > ExecutionEnvironment.
> > > > > > > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > > I'm currently using flink 0.9.0 which by
> maven
> > > > > support
> > > > > > > > > Hadoop
> > > > > > > > > > 1
> > > > > > > > > > > .
> > > > > > > > > > > > > > > > By using
> > > flink-clients-0.7.0-hadoop2-incubating.jar
> > > > > > with
> > > > > > > > > > > > > > executePlan(Plan
> > > > > > > > > > > > > > > > p) method instead, I'm getting the same
> > > exception
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hanan
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. I have restarted Flink service via
> > > > > > > > stop/start-loval.sh -
> > > > > > > > > > it
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > restarted successfully ,no errors in log
> > folder
> > > > > > > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > Failed
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > > > > > > at
> > > > > > > > > >
> org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > >
> org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > >
> org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > > > > > > Caused by: java.io.IOException: JobManager
> at
> > > > > > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> > > > > > :6123/user/jobmanager
> > > > > > > > not
> > > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > > > Please make sure that the JobManager is
> > running
> > > > and
> > > > > > its
> > > > > > > > > port
> > > > > > > > > > is
> > > > > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > >
> org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor
> > not
> > > > > found
> > > > > > > for:
> > > > > > > > > > > > > > > > >
> > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > > > > :6123/),
> > > > > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > >
> > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > > > > > > >
> > > > > > > > > >
> > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > > > > > > at
> > > > > > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > > > > > > at
> > > > > > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > > > > > > at
> > > > > > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > > > > > > at
> > > > > > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > >
> > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > > > > > > at
> > > > > > akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > > > > > > at
> > > > > > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > > > > > > at
> > > > > > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > >
> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > > > > > > at
> > akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > > > > > > at
> > > akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert
> > > Metzger
> > > > <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> I guess you are getting an entire
> exception
> > > > after
> > > > > > the
> > > > > > > > > > > > > > > "org.apache.flink
> > > > > > > > > > > > > > > > >>
> .client.program.ProgramInvocationException:
> > > > Failed
> > > > > > to
> > > > > > > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > > > > > > >> Can you post it here to help us
> > understanding
> > > > the
> > > > > > > issue?
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey
> > > > > Sapozhnikov
> > > > > > <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > Some clarification: locally everything
> > works
> > > > > > great.
> > > > > > > > > > > > > > > > >> > However once we run our Flink on remote
> > > linux
> > > > > > > machine
> > > > > > > > > and
> > > > > > > > > > > try
> > > > > > > > > > > > to
> > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> > client program from our machine, using
> > > create
> > > > > > remote
> > > > > > > > > > > > > environment-
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan
> > > Ewen
> > > > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > > If you start the job via the
> "bin/flink"
> > > > > script,
> > > > > > > > then
> > > > > > > > > > > simply
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > >> > >
> > > > > "ExecutionEnvironment.getExecutionEnvironment()"
> > > > > > > > > rather
> > > > > > > > > > > then
> > > > > > > > > > > > > > > > creating
> > > > > > > > > > > > > > > > >> a
> > > > > > > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > > That way, hosts and ports are
> configured
> > > > > > > > > automatically.
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM,
> Robert
> > > > > Metzger
> > > > > > <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > >> > >> Which values did you use for
> > > > FLINK_SERVER_URL
> > > > > > and
> > > > > > > > > > > > FLINK_PORT?
> > > > > > > > > > > > > > > > >> > >> Every time you deploy Flink on YARN,
> > the
> > > > host
> > > > > > and
> > > > > > > > > port
> > > > > > > > > > > > > change,
> > > > > > > > > > > > > > > > >> because
> > > > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > > > >> > >> JobManager is started on a different
> > YARN
> > > > > > > > container.
> > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM,
> Hanan
> > > > Meyer
> > > > > <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit
> > Flink
> > > to
> > > > > > Yarn
> > > > > > > > > single
> > > > > > > > > > > > node
> > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > >> I'm
> > > > > > > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >
> > > > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > > > > > >> Failed to
> > > > > > > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > The Jar copy a file from one
> location
> > > in
> > > > > Hdfs
> > > > > > > to
> > > > > > > > > > > another
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > >> > fine
> > > > > > > > > > > > > > > > >> > >> > while executed locally on the
> single
> > > node
> > > > > > Yarn
> > > > > > > > > > cluster
> > > > > > > > > > > -
> > > > > > > > > > > > > > > > >> > >> > bin/flink run -c Test
> > > > ./examples/MyJar.jar
> > > > > > > > > > > > > > > > >> > >> > hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > ExecutionEnvironment envRemote
> =
> > > > > > > > > > > > > > > > >> > >> >
> > > > > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > > > > > > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > >
> > > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > >
> data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > *Regards*
> > > > > > > > >
> > > > > > > > > *Alexey Sapozhnikov*
> > > > > > > > > CTO& Co-Founder
> > > > > > > > > Scalabillit Inc
> > > > > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > > > > M : +972-52-2363823
> > > > > > > > > E : [hidden email]
> > > > > > > > > W : http://www.scalabill.it
> > > > > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > > > > Revolutionizing Proof-of-Concept
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *Regards*
> > > > > > >
> > > > > > > *Alexey Sapozhnikov*
> > > > > > > CTO& Co-Founder
> > > > > > > Scalabillit Inc
> > > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > > M : +972-52-2363823
> > > > > > > E : [hidden email]
> > > > > > > W : http://www.scalabill.it
> > > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > > Revolutionizing Proof-of-Concept
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Regards*
> > > > >
> > > > > *Alexey Sapozhnikov*
> > > > > CTO& Co-Founder
> > > > > Scalabillit Inc
> > > > > Aba Even 10-C, Herzelia, Israel
> > > > > M : +972-52-2363823
> > > > > E : [hidden email]
> > > > > W : http://www.scalabill.it
> > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > Map:http://mapta.gs/Scalabillit
> > > > > Revolutionizing Proof-of-Concept
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Regards*
> > >
> > > *Alexey Sapozhnikov*
> > > CTO& Co-Founder
> > > Scalabillit Inc
> > > Aba Even 10-C, Herzelia, Israel
> > > M : +972-52-2363823
> > > E : [hidden email]
> > > W : http://www.scalabill.it
> > > YT - https://youtu.be/9Rj309PTOFA
> > > Map:http://mapta.gs/Scalabillit
> > > Revolutionizing Proof-of-Concept
> > >
> >
>
>
>
> --
>
> *Regards*
>
> *Alexey Sapozhnikov*
> CTO& Co-Founder
> Scalabillit Inc
> Aba Even 10-C, Herzelia, Israel
> M : +972-52-2363823
> E : [hidden email]
> W : http://www.scalabill.it
> YT - https://youtu.be/9Rj309PTOFA
> Map:http://mapta.gs/Scalabillit
> Revolutionizing Proof-of-Concept
>

Alexey Sapozhnikov

Re: Apache Flink:ProgramInvocationException on Yarn

Stephan it is exactly the same exception -UknownHost bal bla
In Jboss for example the external are also not working, only the 0.0.0.0 -
this is AWS NAT.
We will proceed with VPC and then I will update you about what we get.
Thanks for your help.

On Sun, Aug 30, 2015 at 6:05 PM, Stephan Ewen <[hidden email]> wrote:

> Why are the external IPs not working? Any kind of exception you can share?
>
> On Sun, Aug 30, 2015 at 5:02 PM, Alexey Sapozhnikov <[hidden email]>
> wrote:
>
> > it will not help, since the internal IPs are changing in AWS from time to
> > time and you should use only Public IP, which is not recognizable by
> flink.
> > Thats why all app servers, for example JBoss or even Flume are using
> > "0.0.0.0"
> >
> > On Sun, Aug 30, 2015 at 5:53 PM, Stephan Ewen <[hidden email]> wrote:
> >
> > > What you can do as a temporary workaround is to actually enter the IP
> > > address for "jobmanager.rpc.address" - that circumvents the DNS.
> > >
> > > Just saw that Akka 2.4 (released some time in the near future)
> apparently
> > > introduces an option to listen to all network interfaces.
> > >
> > > On Sun, Aug 30, 2015 at 4:44 PM, Alexey Sapozhnikov <
> [hidden email]
> > >
> > > wrote:
> > >
> > > > Fully understand.
> > > > 1.My suggestion is to drop Akka and take something else, since this
> > issue
> > > > is really big
> > > > 2.Not hostname not the endpoint are not working, clarifying the VPC
> > topic
> > > > now.
> > > >
> > > > On Sun, Aug 30, 2015 at 5:41 PM, Stephan Ewen <[hidden email]>
> > wrote:
> > > >
> > > > > Not being able to bind to 0.0.0.0 is an Akka issue. It is sometimes
> > > > > annoying, but I have not found a good way around this.
> > > > > The problem is that the address to bind to an the address used by
> > > others
> > > > to
> > > > > send messages to the node is the same. (
> > > > > https://groups.google.com/forum/#!topic/akka-user/cRZmf8u_vZY)
> > > > >
> > > > >
> > > > > All of that aside, we have been using Flink on Cloud Platforms GCE,
> > > > Amazon,
> > > > > and it worked.
> > > > >
> > > > > Can you try to use the internal hostname, or the fully qualified
> > > external
> > > > > hostname?
> > > > >
> > > > >
> > > > > On Sun, Aug 30, 2015 at 4:23 PM, Alexey Sapozhnikov <
> > > [hidden email]
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi.
> > > > > > First off - many thanks for your efforts and prompt help.
> > > > > > We will try to find how to do it with DNS server on VPC.
> > > > > > however, absence of "0.0.0.0" is definitely a huge bug - just
> think
> > > > about
> > > > > > the current situation : if I dont have a VPC, I cant invoke the
> > Flink
> > > > > > functionality remotely in Amazon.
> > > > > > We will clarify the VPC situation and I will write my impression
> of
> > > > what
> > > > > we
> > > > > > get.
> > > > > >
> > > > > > Thanks, guys
> > > > > >
> > > > > > On Sun, Aug 30, 2015 at 5:16 PM, Robert Metzger <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > From this blog post, it seems that this hostname is not
> > resolvable:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://holtstrom.com/michael/blog/post/401/Hostname-in-Amazon-Linux.html
> > > > > > > Can you easily activate a DNS server in the VPC?
> > > > > > >
> > > > > > > 0.0.0.0 is not supported because of some requirements of the
> Akka
> > > > > > > framework.
> > > > > > > But you should be able to use the host's IP address.
> > > > > > > Can you try that as well?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Aug 30, 2015 at 4:08 PM, Alexey Sapozhnikov <
> > > > > [hidden email]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Here is the exception from the moment we tried to put in
> > > > > > > > jobmanager.rpc.address the hostname of the machine which is
> > > > > > ip-172-36-98
> > > > > > > > looks like it doesnt recognize this address.
> > > > > > > > Why it doesnt support "0.0.0.0"????
> > > > > > > >
> > > > > > > > 13:43:14,805 INFO
> > org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > > -
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> --------------------------------------------------------------------------------
> > > > > > > > 13:43:14,914 INFO
> > org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > > - Loading configuration from
> > > > > > /usr/local/test/flink-0.9.0/bin/../conf
> > > > > > > > 13:43:14,942 INFO
> > org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > > - Security is not enabled. Starting non-authenticated
> > > > > JobManager.
> > > > > > > > 13:43:14,942 INFO
> > org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > > - Starting JobManager
> > > > > > > > 13:43:14,944 INFO
> > org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > > - Starting JobManager actor system at ip-172-36-98:6123.
> > > > > > > > 13:43:15,047 ERROR
> > org.apache.flink.runtime.jobmanager.JobManager
> > > > > > > > - Failed to run JobManager.
> > > > > > > > java.lang.Exception: Could not create JobManager actor system
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:882)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:826)
> > > > > > > > at
> > > > > > > >
> > > > org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> > > > > > > > Caused by: java.net.UnknownHostException: ip-172-36-98:
> unknown
> > > > error
> > > > > > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native
> > > > Method)
> > > > > > > > at
> > > > > > java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > > > > > > > @
> > > > > > > >
> > > > > 21,1
> > > > > > > > 71%
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Aug 30, 2015 at 4:57 PM, Robert Metzger <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > How is Flink crashing when you start it on the Linux
> machine
> > in
> > > > > > Amazon?
> > > > > > > > >
> > > > > > > > > Can you post the exception here?
> > > > > > > > >
> > > > > > > > > On Sun, Aug 30, 2015 at 3:48 PM, Alexey Sapozhnikov <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Stephan.
> > > > > > > > > >
> > > > > > > > > > We run this Linux machine on Amazon, which I predict,
> most
> > of
> > > > the
> > > > > > > > people
> > > > > > > > > > will do.
> > > > > > > > > > We tried to put "0.0.0.0" or Public IP of the machine-
> > Flink
> > > > > > crashes
> > > > > > > on
> > > > > > > > > > start, it doesnt recognize himself.
> > > > > > > > > > It is very strange that it doesnt work with 0.0.0.0-
> > > basically
> > > > > this
> > > > > > > is
> > > > > > > > a
> > > > > > > > > > way in Java to make it being seen widely.
> > > > > > > > > > We tried to put there the hostname - what you get from
> > > > "hostname"
> > > > > > > > > command.
> > > > > > > > > > It crashes.
> > > > > > > > > > It works only with "localhost" and works only locally
> > > > > > > > > > So what you suggest we will put there so the remote
> client
> > > > could
> > > > > > > > connect.
> > > > > > > > > > ?
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > >
> > > > > > > > > > On Sun, Aug 30, 2015 at 4:34 PM, Stephan Ewen <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Do you start Flink via YARN? In that case the
> > > > > > > > "jobmanager.rpc.address"
> > > > > > > > > is
> > > > > > > > > > > not used, because YARN assigns containers/nodes.
> > > > > > > > > > >
> > > > > > > > > > > If you start Flink in "standalone" mode, this should be
> > the
> > > > > > address
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > node that runs the JobManager. It will be used as the
> > > host/IP
> > > > > > that
> > > > > > > > > Flink
> > > > > > > > > > > binds to. The same host should then be used in the
> > > > > > > RemoteEnvironment.
> > > > > > > > > > >
> > > > > > > > > > > Stephan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Aug 30, 2015 at 3:25 PM, Hanan Meyer <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello all.
> > > > > > > > > > > >
> > > > > > > > > > > > Firstly- thank you for your valuable advices.
> > > > > > > > > > > > We did some very fine tuned pinpoint test and comes
> to
> > > > > > following
> > > > > > > > > > > > conclusions
> > > > > > > > > > > >
> > > > > > > > > > > > 1.We run on Ubuntu 14 flink for hadoop 2.7
> > > > > > > > > > > > 2.Once we copy our Java client program directy to the
> > > > machine
> > > > > > and
> > > > > > > > run
> > > > > > > > > > it
> > > > > > > > > > > > directly there it worked very good
> > > > > > > > > > > > The program is
> > > > > > > > > > > >
> > > > > > > > > > > > .....
> > > > > > > > > > > >
> > > > > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > > > >
> > > =ExecutionEnvironment.createRemoteEnvironment("localhost",
> > > > > > 6123,
> > > > > > > > > > > > "\usr\local\HananTestProj.jar");
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > org.apache.flink.api.java.DataSet text =
> > > > > > > > > > > > (org.apache.flink.api.java.DataSet)
> > > envRemote.fromElements(
> > > > > > > > > > > > "Who's there?",
> > > > > > > > > > > > "I think I hear them. Stand, ho! Who's
> > > there?");
> > > > > > > > > > > >
> > > > > > > > > > > >
> org.apache.flink.api.java.DataSet<Tuple2<String,
> > > > > > Integer>>
> > > > > > > > > > > > wordCounts = text
> > > > > > > > > > > > .flatMap(new LineSplitter())
> > > > > > > > > > > > .groupBy(0)
> > > > > > > > > > > > .sum(1);
> > > > > > > > > > > >
> > > > > > > > > > > > wordCounts.print();
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > public static class LineSplitter implements
> > > > > > > > > FlatMapFunction<String,
> > > > > > > > > > > > Tuple2<String, Integer>> {
> > > > > > > > > > > > public void flatMap(String line,
> > > > > > Collector<Tuple2<String,
> > > > > > > > > > > Integer>>
> > > > > > > > > > > > out) {
> > > > > > > > > > > > for (String word : line.split(" ")) {
> > > > > > > > > > > > out.collect(new Tuple2<String,
> > > > Integer>(word,
> > > > > > 1));
> > > > > > > > > > > > }
> > > > > > > > > > > > }
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > .....
> > > > > > > > > > > > Program works fine
> > > > > > > > > > > > 3.Now we are trying to run this program remotely ,
> from
> > > > > windows
> > > > > > > > > machine
> > > > > > > > > > > > when the first row looks differently
> > > > > > > > > > > > ExecutionEnvironment envRemote
> > > > > > > > > > > >
> > =ExecutionEnvironment.createRemoteEnvironment("1.2.3.4",
> > > > > 6123,
> > > > > > > > > > > > "C:\\HananTestProj.jar");
> > > > > > > > > > > > when 1.2.3.4 is IP address of fink machine
> > > > > > > > > > > >
> > > > > > > > > > > > 4.We got an exception :Jobmanager at 1.2.3.4 cant be
> > > > reached
> > > > > > bla
> > > > > > > > bla
> > > > > > > > > > bla
> > > > > > > > > > > >
> > > > > > > > > > > > 5.in flink configuration we found a following line
> > > > > > > > > > > jobmanager.rpc.address:
> > > > > > > > > > > > localhost
> > > > > > > > > > > > Flink cant be started with any other value
> > > > > (hostname/ipaddress
> > > > > > )
> > > > > > > > > except
> > > > > > > > > > > the
> > > > > > > > > > > > localhost
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 6.We assume that probably Flink has a critical bug :
> it
> > > > cant
> > > > > be
> > > > > > > > > started
> > > > > > > > > > > > from remote machine, only locally. Are we right? Are
> we
> > > > > wrong?
> > > > > > > > Should
> > > > > > > > > > we
> > > > > > > > > > > > fill JIRA ?
> > > > > > > > > > > > Maybe we need somehow to configure Flink differently?
> > > > > > > > > > > >
> > > > > > > > > > > > Please advice
> > > > > > > > > > > > Best regards
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Aug 30, 2015 at 3:19 PM, Robert Metzger <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The output of the YARN session should look like
> this:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Flink JobManager is now running on
> > > > > quickstart.cloudera:39956
> > > > > > > > > > > > > JobManager Web Interface:
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> http://quickstart.cloudera:8088/proxy/application_1440768826963_0005/
> > > > > > > > > > > > > Number of connected TaskManagers changed to 1.
> Slots
> > > > > > > available: 1
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Aug 30, 2015 at 11:12 AM, Stephan Ewen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The only thing I can think of is that you are not
> > > using
> > > > > the
> > > > > > > > right
> > > > > > > > > > > > > host/port
> > > > > > > > > > > > > > for the JobManager.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > When you start the YARN session, it should print
> > the
> > > > host
> > > > > > > where
> > > > > > > > > the
> > > > > > > > > > > > > > JobManager runs. You also need to take the port
> > from
> > > > > there,
> > > > > > > as
> > > > > > > > in
> > > > > > > > > > > YARN,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > port is usually not 6123. Yarn starts many
> services
> > > on
> > > > > one
> > > > > > > > > machine,
> > > > > > > > > > > so
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > ports need to be randomized.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It may be worth adding a YARNExecutionEnvironment
> > at
> > > > some
> > > > > > > > point,
> > > > > > > > > > > which
> > > > > > > > > > > > > > deals with this transparent (starting the YARN
> > > cluster,
> > > > > > > > > connecting
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > JobManager).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Aug 30, 2015 at 10:12 AM, Hanan Meyer <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hello.
> > > > > > > > > > > > > > > Let me clarify the situation.
> > > > > > > > > > > > > > > 1. We are using flink 0.9.0 for Hadoop 2.7. We
> > > > > connected
> > > > > > it
> > > > > > > > to
> > > > > > > > > > HDFS
> > > > > > > > > > > > > > 2.7.1.
> > > > > > > > > > > > > > > 2. Locally, our program is working: once we run
> > > flink
> > > > > as
> > > > > > > > > > > > > > ./start-local.sh,
> > > > > > > > > > > > > > > we are able to connect and run the
> > > > > > createRemoteEnvironment
> > > > > > > > and
> > > > > > > > > > > > Execute
> > > > > > > > > > > > > > > methods.
> > > > > > > > > > > > > > > 3.Due to our architecture and basic Flink
> feature
> > > we
> > > > > want
> > > > > > > to
> > > > > > > > > > invoke
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > functionality REMOTELY , when our Java code is
> > > > calling
> > > > > > the
> > > > > > > > > Flink
> > > > > > > > > > > > > methods
> > > > > > > > > > > > > > > from another server.
> > > > > > > > > > > > > > > 4.We tried both
> > > > > > > > > > > >
> ExecutionEnvironment.createRemoteEnvironment("1.2.3.1",
> > > > > > > > > > > > > > > 6123, "TestProj.jar"); and
> > > > > > > > > > > > > ExecutionEnvironment.createRemoteEnvironment("
> > > > > > > > > > > > > > > flink@1.2.3.1", 6123, "TestProj.jar"); (which
> is
> > > > > > > definitely
> > > > > > > > > not
> > > > > > > > > > > > right
> > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > it should be an IP address) - it crash on the
> > "cant
> > > > > reach
> > > > > > > > > > > JobManager"
> > > > > > > > > > > > > > > error.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It seems to us that it can be one of 2 issues.
> > > > > > > > > > > > > > > 1.Somehow we need to configure flink to accept
> > the
> > > > > > > > connections
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > remote machine
> > > > > > > > > > > > > > > 2.Flink has a critical showstopper bug that
> > > > > jeopardizing
> > > > > > a
> > > > > > > > > whole
> > > > > > > > > > > > > decision
> > > > > > > > > > > > > > > to use this technology.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please advise us how we should advance.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 10:27 AM, Robert
> Metzger
> > <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > in the exception you've posted earlier, you
> can
> > > see
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > > root
> > > > > > > > > > > > > > > > cause:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound: Actor
> not
> > > > found
> > > > > > for:
> > > > > > > > > > > > > > > >
> > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > > > :6123/),
> > > > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This string
> "akka.tcp://flink@FLINK_SERVER_URL
> > > > > :6123/"
> > > > > > > > > usually
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > this: "akka.tcp://flink@1.2.3.4:6123/". So
> it
> > > > seems
> > > > > > that
> > > > > > > > you
> > > > > > > > > > are
> > > > > > > > > > > > > > > > passing FLINK_SERVER_URL
> > > > > > > > > > > > > > > > as the server hostname (or ip).
> > > > > > > > > > > > > > > > Can you pass the correct hostname when you
> call
> > > > > > > > > > > > ExecutionEnvironment.
> > > > > > > > > > > > > > > > createRemoteEnvironment().
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 7:52 AM, Hanan Meyer
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > > > I'm currently using flink 0.9.0 which by
> > maven
> > > > > > support
> > > > > > > > > > Hadoop
> > > > > > > > > > > 1
> > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > By using
> > > > flink-clients-0.7.0-hadoop2-incubating.jar
> > > > > > > with
> > > > > > > > > > > > > > > executePlan(Plan
> > > > > > > > > > > > > > > > > p) method instead, I'm getting the same
> > > > exception
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hanan
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 8:35 AM, Hanan
> Meyer
> > <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. I have restarted Flink service via
> > > > > > > > > stop/start-loval.sh -
> > > > > > > > > > > it
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > restarted successfully ,no errors in log
> > > folder
> > > > > > > > > > > > > > > > > > 2. default flink port is -6123
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Getting this via Eclips IDE:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > > Failed
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > resolve JobManager
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > org.apache.flink.client.program.Client.run(Client.java:379)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > org.apache.flink.client.program.Client.run(Client.java:356)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > org.apache.flink.client.program.Client.run(Client.java:349)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> > > > > > > > > > > > > > > > > > at Test.main(Test.java:39)
> > > > > > > > > > > > > > > > > > Caused by: java.io.IOException:
> JobManager
> > at
> > > > > > > > > > > > > > > > > > akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > :6123/user/jobmanager
> > > > > > > > > not
> > > > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > > > > Please make sure that the JobManager is
> > > running
> > > > > and
> > > > > > > its
> > > > > > > > > > port
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > > reachable.
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > org.apache.flink.client.program.Client.run(Client.java:376)
> > > > > > > > > > > > > > > > > > ... 7 more
> > > > > > > > > > > > > > > > > > Caused by: akka.actor.ActorNotFound:
> Actor
> > > not
> > > > > > found
> > > > > > > > for:
> > > > > > > > > > > > > > > > > >
> > > > > > > ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL
> > > > > > > > > > > > :6123/),
> > > > > > > > > > > > > > > > > > Path(/user/jobmanager)]
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> > > > > > > > > > > > > > > > > > at akka.dispatch.BatchingExecutor$
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > >
> > > akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> > > > > > > > > > > > > > > > > > at
> > > > > > > akka.actor.ActorCell.terminate(ActorCell.scala:369)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > >
> > > > > > > >
> > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> > > > > > > > > > > > > > > > > > at
> > > akka.dispatch.Mailbox.run(Mailbox.scala:220)
> > > > > > > > > > > > > > > > > > at
> > > > akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Aug 27, 2015 at 10:47 PM, Robert
> > > > Metzger
> > > > > <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> I guess you are getting an entire
> > exception
> > > > > after
> > > > > > > the
> > > > > > > > > > > > > > > > "org.apache.flink
> > > > > > > > > > > > > > > > > >>
> > .client.program.ProgramInvocationException:
> > > > > Failed
> > > > > > > to
> > > > > > > > > > > > > > > > > >> resolve JobManager".
> > > > > > > > > > > > > > > > > >> Can you post it here to help us
> > > understanding
> > > > > the
> > > > > > > > issue?
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> On Thu, Aug 27, 2015 at 6:55 PM, Alexey
> > > > > > Sapozhnikov
> > > > > > > <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> > Hello all.
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > Some clarification: locally everything
> > > works
> > > > > > > great.
> > > > > > > > > > > > > > > > > >> > However once we run our Flink on
> remote
> > > > linux
> > > > > > > > machine
> > > > > > > > > > and
> > > > > > > > > > > > try
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> > client program from our machine, using
> > > > create
> > > > > > > remote
> > > > > > > > > > > > > > environment-
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > >> > JobManager is raising this exception
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > On Thu, Aug 27, 2015 at 7:41 PM,
> Stephan
> > > > Ewen
> > > > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >> > > If you start the job via the
> > "bin/flink"
> > > > > > script,
> > > > > > > > > then
> > > > > > > > > > > > simply
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > >> > >
> > > > > > "ExecutionEnvironment.getExecutionEnvironment()"
> > > > > > > > > > rather
> > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > creating
> > > > > > > > > > > > > > > > > >> a
> > > > > > > > > > > > > > > > > >> > > remote environment manually.
> > > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > > >> > > That way, hosts and ports are
> > configured
> > > > > > > > > > automatically.
> > > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > > >> > > On Thu, Aug 27, 2015 at 6:39 PM,
> > Robert
> > > > > > Metzger
> > > > > > > <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > > >> > >> Hi,
> > > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > > >> > >> Which values did you use for
> > > > > FLINK_SERVER_URL
> > > > > > > and
> > > > > > > > > > > > > FLINK_PORT?
> > > > > > > > > > > > > > > > > >> > >> Every time you deploy Flink on
> YARN,
> > > the
> > > > > host
> > > > > > > and
> > > > > > > > > > port
> > > > > > > > > > > > > > change,
> > > > > > > > > > > > > > > > > >> because
> > > > > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > > > > >> > >> JobManager is started on a
> different
> > > YARN
> > > > > > > > > container.
> > > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > > >> > >> On Thu, Aug 27, 2015 at 6:32 PM,
> > Hanan
> > > > > Meyer
> > > > > > <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > > >> > >> > Hello All
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > When using Eclipse IDE to submit
> > > Flink
> > > > to
> > > > > > > Yarn
> > > > > > > > > > single
> > > > > > > > > > > > > node
> > > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > >> I'm
> > > > > > > > > > > > > > > > > >> > >> > getting :
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >
> > > > > > "org.apache.flink.client.program.ProgramInvocationException:
> > > > > > > > > > > > > > > > > >> Failed to
> > > > > > > > > > > > > > > > > >> > >> > resolve JobManager"
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > Using Flink 0.9.0
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > The Jar copy a file from one
> > location
> > > > in
> > > > > > Hdfs
> > > > > > > > to
> > > > > > > > > > > > another
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > >> > fine
> > > > > > > > > > > > > > > > > >> > >> > while executed locally on the
> > single
> > > > node
> > > > > > > Yarn
> > > > > > > > > > > cluster
> > > > > > > > > > > > -
> > > > > > > > > > > > > > > > > >> > >> > bin/flink run -c Test
> > > > > ./examples/MyJar.jar
> > > > > > > > > > > > > > > > > >> > >> >
> hdfs://localhost:9000/flink/in.txt
> > > > > > > > > > > > > > > > > >> hdfs://localhost:9000/flink/out.txt
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > The code skeleton:
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > ExecutionEnvironment
> envRemote
> > =
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > ExecutionEnvironment.createRemoteEnvironment
> > > > > > > > > > > > > > > > > >> > >> > (FLINK_SERVER_URL,FLINK
> > > > > > > > PORT,JAR_PATH_ON_CLIENT);
> > > > > > > > > > > > > > > > > >> > >> > DataSet<String> data =
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > >
> > > > > > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > >
> > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > > > > > > > > > > > > > > > > >> > >> > envRemote.execute();
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > Please advise,
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >> > Hanan Meyer
> > > > > > > > > > > > > > > > > >> > >> >
> > > > > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > *Regards*
> > > > > > > > > >
> > > > > > > > > > *Alexey Sapozhnikov*
> > > > > > > > > > CTO& Co-Founder
> > > > > > > > > > Scalabillit Inc
> > > > > > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > > > > > M : +972-52-2363823
> > > > > > > > > > E : [hidden email]
> > > > > > > > > > W : http://www.scalabill.it
> > > > > > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > > > > > Revolutionizing Proof-of-Concept
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > *Regards*
> > > > > > > >
> > > > > > > > *Alexey Sapozhnikov*
> > > > > > > > CTO& Co-Founder
> > > > > > > > Scalabillit Inc
> > > > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > > > M : +972-52-2363823
> > > > > > > > E : [hidden email]
> > > > > > > > W : http://www.scalabill.it
> > > > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > > > Revolutionizing Proof-of-Concept
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Regards*
> > > > > >
> > > > > > *Alexey Sapozhnikov*
> > > > > > CTO& Co-Founder
> > > > > > Scalabillit Inc
> > > > > > Aba Even 10-C, Herzelia, Israel
> > > > > > M : +972-52-2363823
> > > > > > E : [hidden email]
> > > > > > W : http://www.scalabill.it
> > > > > > YT - https://youtu.be/9Rj309PTOFA
> > > > > > Map:http://mapta.gs/Scalabillit
> > > > > > Revolutionizing Proof-of-Concept
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Regards*
> > > >
> > > > *Alexey Sapozhnikov*
> > > > CTO& Co-Founder
> > > > Scalabillit Inc
> > > > Aba Even 10-C, Herzelia, Israel
> > > > M : +972-52-2363823
> > > > E : [hidden email]
> > > > W : http://www.scalabill.it
> > > > YT - https://youtu.be/9Rj309PTOFA
> > > > Map:http://mapta.gs/Scalabillit
> > > > Revolutionizing Proof-of-Concept
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Regards*
> >
> > *Alexey Sapozhnikov*
> > CTO& Co-Founder
> > Scalabillit Inc
> > Aba Even 10-C, Herzelia, Israel
> > M : +972-52-2363823
> > E : [hidden email]
> > W : http://www.scalabill.it
> > YT - https://youtu.be/9Rj309PTOFA
> > Map:http://mapta.gs/Scalabillit
> > Revolutionizing Proof-of-Concept
> >
>