Hi everybody,
I'm Ned, a young and passionte developer of apache technologies. I have been playing with apache flink lastly. This is what I wanted to do submit a flink topology to a remote flink cluster. The following are the steps that I did. - Install flink as a cluster indicated on the link https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html on three remotes VMs. - Run the sample WordCountRemoteByClient <https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java> by changing conf.put(Config.NIMBUS_HOST, "localhost"); to conf.put(Config.NIMBUS_HOST, "publicIpOfJobmanagerInMyRemoteCluster"); Unfortunately for me when I run that program, I have a the following exception. org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) at org.apache.flink.client.program.Client.runBlocking(Client.java:248) at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) Caused by: java.lang.RuntimeException: Could not connect to Flink JobManager with address publicIpOfJobmanagerInMyRemoteCluster:6123 at org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) at org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) at org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) at stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) ... 6 more Caused by: java.io.IOException: Actor at akka.tcp://flink@ publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not reachable. Please make sure that the actor is running and its port is reachable. at org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) at org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) at org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) at org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) at org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) at org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) ... 14 more Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) ... 19 more I try ping my jobmanager with curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the following as responces. curl: (52) Empty reply from server Which is an indication that the job manager is reachable. So I was wondering if I doing it the right way. Please any help will be welcoming. Thanks, Ned |
Hi Ned,
what does the logs of the JobManager say? Cheers, Till On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: > Hi everybody, > > I'm Ned, a young and passionte developer of apache technologies. I have > been playing with apache flink lastly. > > This is what I wanted to do submit a flink topology to a remote flink > cluster. The following are the steps that I did. > > - Install flink as a cluster indicated on the link > > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > on three remotes VMs. > - Run the sample WordCountRemoteByClient > < > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > > > by > changing > conf.put(Config.NIMBUS_HOST, "localhost"); to > conf.put(Config.NIMBUS_HOST, "publicIpOfJobmanagerInMyRemoteCluster"); > > Unfortunately for me when I run that program, I have a the following > exception. > > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error. > at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > at > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > at org.apache.flink.client.program.Client.runBlocking(Client.java:248) > at > > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > Caused by: java.lang.RuntimeException: Could not connect to Flink > JobManager with address publicIpOfJobmanagerInMyRemoteCluster:6123 > at > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > at > > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > at > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > at > > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > ... 6 more > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not reachable. > Please make sure that the actor is running and its port is reachable. > at > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > at > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > at > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > at > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > at > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > at > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > ... 14 more > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [10000 milliseconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > ... 19 more > > I try ping my jobmanager with > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the following as > responces. > > curl: (52) Empty reply from server > > Which is an indication that the job manager is reachable. > > So I was wondering if I doing it the right way. Please any help will be > welcoming. > > Thanks, > Ned > |
Hi Till
Thanks for the prompt reply. The logs say that Please make sure that the actor is running and its port is reachable. And it is actaully reachable because I can ping that address. Ned. On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann <[hidden email]> wrote: > Hi Ned, > > what does the logs of the JobManager say? > > Cheers, > Till > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: > > > Hi everybody, > > > > I'm Ned, a young and passionte developer of apache technologies. I have > > been playing with apache flink lastly. > > > > This is what I wanted to do submit a flink topology to a remote flink > > cluster. The following are the steps that I did. > > > > - Install flink as a cluster indicated on the link > > > > > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > > on three remotes VMs. > > - Run the sample WordCountRemoteByClient > > < > > > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > > > > > by > > changing > > conf.put(Config.NIMBUS_HOST, "localhost"); to > > conf.put(Config.NIMBUS_HOST, "publicIpOfJobmanagerInMyRemoteCluster"); > > > > Unfortunately for me when I run that program, I have a the following > > exception. > > > > org.apache.flink.client.program.ProgramInvocationException: The main > method > > caused an error. > > at > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > > at > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > > at org.apache.flink.client.program.Client.runBlocking(Client.java:248) > > at > > > > > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > > at > > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > > Caused by: java.lang.RuntimeException: Could not connect to Flink > > JobManager with address publicIpOfJobmanagerInMyRemoteCluster:6123 > > at > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > > at > > > > > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > > at > > > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > > at > > > > > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:483) > > at > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > > ... 6 more > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not reachable. > > Please make sure that the actor is running and its port is reachable. > > at > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > > at > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > > at > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > > at > > > > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > > at > > > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > > at > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > > ... 14 more > > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > > [10000 milliseconds] > > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > > > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > > ... 19 more > > > > I try ping my jobmanager with > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the following as > > responces. > > > > curl: (52) Empty reply from server > > > > Which is an indication that the job manager is reachable. > > > > So I was wondering if I doing it the right way. Please any help will be > > welcoming. > > > > Thanks, > > Ned > > > |
I'm referring to the jobmanager.log file not the client log file. You can
find it in the `/log` directory. Cheers, Till On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> wrote: > Hi Till > > Thanks for the prompt reply. > > The logs say that Please make sure that the actor is running and its port > is reachable. > And it is actaully reachable because I can ping that address. > > Ned. > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Ned, > > > > what does the logs of the JobManager say? > > > > Cheers, > > Till > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: > > > > > Hi everybody, > > > > > > I'm Ned, a young and passionte developer of apache technologies. I have > > > been playing with apache flink lastly. > > > > > > This is what I wanted to do submit a flink topology to a remote flink > > > cluster. The following are the steps that I did. > > > > > > - Install flink as a cluster indicated on the link > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > > > on three remotes VMs. > > > - Run the sample WordCountRemoteByClient > > > < > > > > > > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > > > > > > > by > > > changing > > > conf.put(Config.NIMBUS_HOST, "localhost"); to > > > conf.put(Config.NIMBUS_HOST, "publicIpOfJobmanagerInMyRemoteCluster"); > > > > > > Unfortunately for me when I run that program, I have a the following > > > exception. > > > > > > org.apache.flink.client.program.ProgramInvocationException: The main > > method > > > caused an error. > > > at > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > > > at > > > > > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > > > at org.apache.flink.client.program.Client.runBlocking(Client.java:248) > > > at > > > > > > > > > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > > > at > > > > > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > > > Caused by: java.lang.RuntimeException: Could not connect to Flink > > > JobManager with address publicIpOfJobmanagerInMyRemoteCluster:6123 > > > at > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > > > at > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > > > at > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > > > at > > > > > > > > > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > at > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:483) > > > at > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > > > ... 6 more > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not > reachable. > > > Please make sure that the actor is running and its port is reachable. > > > at > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > > > at > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > > > at > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > > > at > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > > > at > > > > > > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > > > at > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > > > ... 14 more > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after > > > [10000 milliseconds] > > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > > > > > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > > > ... 19 more > > > > > > I try ping my jobmanager with > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the following as > > > responces. > > > > > > curl: (52) Empty reply from server > > > > > > Which is an indication that the job manager is reachable. > > > > > > So I was wondering if I doing it the right way. Please any help will be > > > welcoming. > > > > > > Thanks, > > > Ned > > > > > > |
2016-04-14 08:23:51,900 INFO
org.apache.flink.runtime.jobmanager.JobManager - -------------------------------------------------------------------------------- 2016-04-14 08:23:51,902 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) 2016-04-14 08:23:51,902 INFO org.apache.flink.runtime.jobmanager.JobManager - Current user: root 2016-04-14 08:23:51,902 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 2016-04-14 08:23:51,902 INFO org.apache.flink.runtime.jobmanager.JobManager - Maximum heap size: 247 MiBytes 2016-04-14 08:23:51,902 INFO org.apache.flink.runtime.jobmanager.JobManager - JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 2016-04-14 08:23:51,929 INFO org.apache.flink.runtime.jobmanager.JobManager - Hadoop version: 1.2.1 2016-04-14 08:23:51,929 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM Options: 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xms256m 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xmx256m 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -XX:MaxPermSize=256m 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - Program Arguments: 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - --configDir 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - /home/ubuntu/flink/conf 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - --executionMode 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - cluster 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - Classpath: /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: 2016-04-14 08:23:51,930 INFO org.apache.flink.runtime.jobmanager.JobManager - -------------------------------------------------------------------------------- 2016-04-14 08:23:51,931 INFO org.apache.flink.runtime.jobmanager.JobManager - Registered UNIX signal handlers for [TERM, HUP, INT] 2016-04-14 08:23:52,362 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /home/ubuntu/flink/conf 2016-04-14 08:23:52,400 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager without high-availability 2016-04-14 08:23:52,408 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on 172.31.45.232:6123 with execution mode CLUSTER 2016-04-14 08:23:52,655 INFO org.apache.flink.runtime.jobmanager.JobManager - Security is not enabled. Starting non-authenticated JobManager. 2016-04-14 08:23:52,701 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager 2016-04-14 08:23:52,701 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system at 172.31.45.232:6123 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2016-04-14 08:23:54,293 INFO Remoting - Starting remoting 2016-04-14 08:23:54,712 INFO Remoting - Remoting started; listening on addresses :[akka.tcp:// flink@172.31.45.232:6123] 2016-04-14 08:23:54,732 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManger web frontend 2016-04-14 08:23:54,777 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log 2016-04-14 08:23:54,777 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out 2016-04-14 08:23:54,805 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the web interface files 2016-04-14 08:23:54,805 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 for web frontend JAR file uploads 2016-04-14 08:23:55,600 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0:0:0:0:0:0:0:0:8081 2016-04-14 08:23:55,601 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor 2016-04-14 08:23:55,619 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 2016-04-14 08:23:55,634 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:60439 - max concurrent requests: 50 - max backlog: 1000 2016-04-14 08:23:55,653 INFO org.apache.flink.runtime.checkpoint.SavepointStoreFactory - Using job manager savepoint state backend. 2016-04-14 08:23:55,678 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager on port 8081 2016-04-14 08:23:55,678 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink@172.31.45.232:6123/user/jobmanager:null. 2016-04-14 08:23:55,692 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. 2016-04-14 08:23:55,696 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/archive 2016-04-14 08:23:55,702 INFO org.apache.flink.runtime.jobmanager.JobManager - JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted leadership with leader session ID None. 2016-04-14 08:24:12,740 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at tresor-testflinkth (akka.tcp:// flink@172.31.45.130:42189/user/taskmanager) as 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts is 1. Current number of alive task slots is 1. 2016-04-14 08:24:17,855 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at tresor-testflinkth (akka.tcp:// flink@172.31.34.121:58814/user/taskmanager) as fead50f6831aa3f341d58162bb918d90. Current number of registered hosts is 2. Current number of alive task slots is 2. 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp:// flink@54.233.183.228:6123/]] arriving at [akka.tcp:// flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// flink@172.31.45.232:6123] 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp:// flink@127.0.0.1:35953] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the public address of the VM hosting job manager and 172.31.45.232 <http://flink@172.31.45.232:6123> is it private address. 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> and 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> are the private address of the task manager On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email]> wrote: > I'm referring to the jobmanager.log file not the client log file. You can > find it in the `/log` directory. > > Cheers, > Till > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> wrote: > > > Hi Till > > > > Thanks for the prompt reply. > > > > The logs say that Please make sure that the actor is running and its port > > is reachable. > > And it is actaully reachable because I can ping that address. > > > > Ned. > > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Ned, > > > > > > what does the logs of the JobManager say? > > > > > > Cheers, > > > Till > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: > > > > > > > Hi everybody, > > > > > > > > I'm Ned, a young and passionte developer of apache technologies. I > have > > > > been playing with apache flink lastly. > > > > > > > > This is what I wanted to do submit a flink topology to a remote flink > > > > cluster. The following are the steps that I did. > > > > > > > > - Install flink as a cluster indicated on the link > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > > > > on three remotes VMs. > > > > - Run the sample WordCountRemoteByClient > > > > < > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > > > > > > > > > by > > > > changing > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to > > > > conf.put(Config.NIMBUS_HOST, > "publicIpOfJobmanagerInMyRemoteCluster"); > > > > > > > > Unfortunately for me when I run that program, I have a the following > > > > exception. > > > > > > > > org.apache.flink.client.program.ProgramInvocationException: The main > > > method > > > > caused an error. > > > > at > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > > > > at > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > > > > at > org.apache.flink.client.program.Client.runBlocking(Client.java:248) > > > > at > > > > > > > > > > > > > > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > > > > at > > > > > > > > > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > > > > Caused by: java.lang.RuntimeException: Could not connect to Flink > > > > JobManager with address publicIpOfJobmanagerInMyRemoteCluster:6123 > > > > at > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > > > > at > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > > > > at > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > > > > at > > > > > > > > > > > > > > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > at > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:483) > > > > at > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > > > > ... 6 more > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not > > reachable. > > > > Please make sure that the actor is running and its port is reachable. > > > > at > > > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > > > > at > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > > > > at > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > > > > at > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > > > > at > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > > > > at > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > > > > ... 14 more > > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after > > > > [10000 milliseconds] > > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > > at > > > > > > > > > > > > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > > at scala.concurrent.Await$.result(package.scala:107) > > > > at > > > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > > > > ... 19 more > > > > > > > > I try ping my jobmanager with > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the following > as > > > > responces. > > > > > > > > curl: (52) Empty reply from server > > > > > > > > Which is an indication that the job manager is reachable. > > > > > > > > So I was wondering if I doing it the right way. Please any help will > be > > > > welcoming. > > > > > > > > Thanks, > > > > Ned > > > > > > > > > > |
Hi Ned,
I think you are facing the issue described in this JIRA issue [1]. The problem is that you have a private and a public IP address and that Akka binds to the private IP address. Since the registered IP of an ActorSystem and the target IP address of a request to this ActorSystem have to be matching, you cannot reach the ActorSystem via the public IP address. Requests with a non-matching IP address are discarded, as indicated by the last log statements. [1] https://issues.apache.org/jira/browse/FLINK-2821 Cheers, Till On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> wrote: > 2016-04-14 08:23:51,900 INFO > org.apache.flink.runtime.jobmanager.JobManager - > > -------------------------------------------------------------------------------- > 2016-04-14 08:23:51,902 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) > 2016-04-14 08:23:51,902 INFO > org.apache.flink.runtime.jobmanager.JobManager - Current > user: root > 2016-04-14 08:23:51,902 INFO > org.apache.flink.runtime.jobmanager.JobManager - JVM: > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 > 2016-04-14 08:23:51,902 INFO > org.apache.flink.runtime.jobmanager.JobManager - Maximum > heap size: 247 MiBytes > 2016-04-14 08:23:51,902 INFO > org.apache.flink.runtime.jobmanager.JobManager - > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 > 2016-04-14 08:23:51,929 INFO > org.apache.flink.runtime.jobmanager.JobManager - Hadoop > version: 1.2.1 > 2016-04-14 08:23:51,929 INFO > org.apache.flink.runtime.jobmanager.JobManager - JVM > Options: > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Xms256m > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Xmx256m > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -XX:MaxPermSize=256m > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > > -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - Program > Arguments: > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > --configDir > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > /home/ubuntu/flink/conf > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > --executionMode > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > cluster > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > Classpath: > > /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: > 2016-04-14 08:23:51,930 INFO > org.apache.flink.runtime.jobmanager.JobManager - > > -------------------------------------------------------------------------------- > 2016-04-14 08:23:51,931 INFO > org.apache.flink.runtime.jobmanager.JobManager - Registered > UNIX signal handlers for [TERM, HUP, INT] > 2016-04-14 08:23:52,362 INFO > org.apache.flink.runtime.jobmanager.JobManager - Loading > configuration from /home/ubuntu/flink/conf > 2016-04-14 08:23:52,400 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager without high-availability > 2016-04-14 08:23:52,408 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager on 172.31.45.232:6123 with execution mode CLUSTER > 2016-04-14 08:23:52,655 INFO > org.apache.flink.runtime.jobmanager.JobManager - Security > is not enabled. Starting non-authenticated JobManager. > 2016-04-14 08:23:52,701 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager > 2016-04-14 08:23:52,701 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager actor system at 172.31.45.232:6123 > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger > - Slf4jLogger started > 2016-04-14 08:23:54,293 INFO Remoting > - Starting remoting > 2016-04-14 08:23:54,712 INFO Remoting > - Remoting started; listening on addresses :[akka.tcp:// > flink@172.31.45.232:6123] > 2016-04-14 08:23:54,732 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManger web frontend > 2016-04-14 08:23:54,777 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of JobManager log file: > > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > 2016-04-14 08:23:54,777 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of JobManager stdout file: > > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out > 2016-04-14 08:23:54,805 INFO > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the web > interface files > 2016-04-14 08:23:54,805 INFO > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 for > web frontend JAR file uploads > 2016-04-14 08:23:55,600 INFO > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web > frontend listening at 0:0:0:0:0:0:0:0:8081 > 2016-04-14 08:23:55,601 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager actor > 2016-04-14 08:23:55,619 INFO org.apache.flink.runtime.blob.BlobServer > - Created BLOB server storage directory > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 > 2016-04-14 08:23:55,634 INFO org.apache.flink.runtime.blob.BlobServer > - Started BLOB server at 0.0.0.0:60439 - max concurrent > requests: 50 - max backlog: 1000 > 2016-04-14 08:23:55,653 INFO > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - Using job > manager savepoint state backend. > 2016-04-14 08:23:55,678 INFO > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager on > port > 8081 > 2016-04-14 08:23:55,678 INFO > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader > reachable under akka.tcp://flink@172.31.45.232:6123/user/jobmanager:null. > 2016-04-14 08:23:55,692 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. > 2016-04-14 08:23:55,696 INFO > org.apache.flink.runtime.jobmanager.MemoryArchivist - Started > memory archivist akka://flink/user/archive > 2016-04-14 08:23:55,702 INFO > org.apache.flink.runtime.jobmanager.JobManager - JobManager > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted leadership > with leader session ID None. > 2016-04-14 08:24:12,740 INFO > org.apache.flink.runtime.instance.InstanceManager - Registered > TaskManager at tresor-testflinkth (akka.tcp:// > flink@172.31.45.130:42189/user/taskmanager) as > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts is 1. > Current number of alive task slots is 1. > 2016-04-14 08:24:17,855 INFO > org.apache.flink.runtime.instance.InstanceManager - Registered > TaskManager at tresor-testflinkth (akka.tcp:// > flink@172.31.34.121:58814/user/taskmanager) as > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts is 2. > Current number of alive task slots is 2. > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter > - dropping message [class > akka.actor.ActorSelectionMessage] for non-local recipient > [Actor[akka.tcp:// > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// > flink@172.31.45.232:6123] > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > flink@127.0.0.1:35953] has failed, address is now gated for [5000] ms. > Reason is: [Disassociated]. > > > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the public > address of the VM hosting job manager and 172.31.45.232 > <http://flink@172.31.45.232:6123> is it private address. > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> and > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> are the > private address of the task manager > > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email]> > wrote: > > > I'm referring to the jobmanager.log file not the client log file. You can > > find it in the `/log` directory. > > > > Cheers, > > Till > > > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> wrote: > > > > > Hi Till > > > > > > Thanks for the prompt reply. > > > > > > The logs say that Please make sure that the actor is running and its > port > > > is reachable. > > > And it is actaully reachable because I can ping that address. > > > > > > Ned. > > > > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < > [hidden email]> > > > wrote: > > > > > > > Hi Ned, > > > > > > > > what does the logs of the JobManager say? > > > > > > > > Cheers, > > > > Till > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: > > > > > > > > > Hi everybody, > > > > > > > > > > I'm Ned, a young and passionte developer of apache technologies. I > > have > > > > > been playing with apache flink lastly. > > > > > > > > > > This is what I wanted to do submit a flink topology to a remote > flink > > > > > cluster. The following are the steps that I did. > > > > > > > > > > - Install flink as a cluster indicated on the link > > > > > > > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > > > > > on three remotes VMs. > > > > > - Run the sample WordCountRemoteByClient > > > > > < > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > > > > > > > > > > > by > > > > > changing > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to > > > > > conf.put(Config.NIMBUS_HOST, > > "publicIpOfJobmanagerInMyRemoteCluster"); > > > > > > > > > > Unfortunately for me when I run that program, I have a the > following > > > > > exception. > > > > > > > > > > org.apache.flink.client.program.ProgramInvocationException: The > main > > > > method > > > > > caused an error. > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > > > > > at > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > > > > > at > > > > > > > > > > > > > > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > > > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > > > > > Caused by: java.lang.RuntimeException: Could not connect to Flink > > > > > JobManager with address publicIpOfJobmanagerInMyRemoteCluster:6123 > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > > > > > at > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > > > > > at > > > > > > > > > > > > > > > > > > > > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > at > > > > > > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > > at > > > > > > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > at java.lang.reflect.Method.invoke(Method.java:483) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > > > > > ... 6 more > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not > > > reachable. > > > > > Please make sure that the actor is running and its port is > reachable. > > > > > at > > > > > > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > > > > > at > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > > > > > ... 14 more > > > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > > after > > > > > [10000 milliseconds] > > > > > at > > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > > > at > > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > > > at > > > > > > > > > > > > > > > > > > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > > > at scala.concurrent.Await$.result(package.scala:107) > > > > > at > > > > > > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > > > > > ... 19 more > > > > > > > > > > I try ping my jobmanager with > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the following > > as > > > > > responces. > > > > > > > > > > curl: (52) Empty reply from server > > > > > > > > > > Which is an indication that the job manager is reachable. > > > > > > > > > > So I was wondering if I doing it the right way. Please any help > will > > be > > > > > welcoming. > > > > > > > > > > Thanks, > > > > > Ned > > > > > > > > > > > > > > > |
Thanks Till for the reply.
But according to you how can I address that? Thanks, Ned On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann <[hidden email]> wrote: > Hi Ned, > > I think you are facing the issue described in this JIRA issue [1]. The > problem is that you have a private and a public IP address and that Akka > binds to the private IP address. Since the registered IP of an ActorSystem > and the target IP address of a request to this ActorSystem have to be > matching, you cannot reach the ActorSystem via the public IP address. > Requests with a non-matching IP address are discarded, as indicated by the > last log statements. > > [1] https://issues.apache.org/jira/browse/FLINK-2821 > > Cheers, > Till > > On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> wrote: > > > 2016-04-14 08:23:51,900 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > > > > -------------------------------------------------------------------------------- > > 2016-04-14 08:23:51,902 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > Starting > > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) > > 2016-04-14 08:23:51,902 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Current > > user: root > > 2016-04-14 08:23:51,902 INFO > > org.apache.flink.runtime.jobmanager.JobManager - JVM: > > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 > > 2016-04-14 08:23:51,902 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Maximum > > heap size: 247 MiBytes > > 2016-04-14 08:23:51,902 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 > > 2016-04-14 08:23:51,929 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Hadoop > > version: 1.2.1 > > 2016-04-14 08:23:51,929 INFO > > org.apache.flink.runtime.jobmanager.JobManager - JVM > > Options: > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > -Xms256m > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > -Xmx256m > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > -XX:MaxPermSize=256m > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > > > > -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Program > > Arguments: > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > --configDir > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > /home/ubuntu/flink/conf > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > --executionMode > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > cluster > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > Classpath: > > > > > /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: > > 2016-04-14 08:23:51,930 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > > > > > -------------------------------------------------------------------------------- > > 2016-04-14 08:23:51,931 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > Registered > > UNIX signal handlers for [TERM, HUP, INT] > > 2016-04-14 08:23:52,362 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Loading > > configuration from /home/ubuntu/flink/conf > > 2016-04-14 08:23:52,400 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManager without high-availability > > 2016-04-14 08:23:52,408 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManager on 172.31.45.232:6123 with execution mode CLUSTER > > 2016-04-14 08:23:52,655 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Security > > is not enabled. Starting non-authenticated JobManager. > > 2016-04-14 08:23:52,701 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManager > > 2016-04-14 08:23:52,701 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManager actor system at 172.31.45.232:6123 > > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger > > - Slf4jLogger started > > 2016-04-14 08:23:54,293 INFO Remoting > > - Starting remoting > > 2016-04-14 08:23:54,712 INFO Remoting > > - Remoting started; listening on addresses :[akka.tcp:// > > flink@172.31.45.232:6123] > > 2016-04-14 08:23:54,732 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManger web frontend > > 2016-04-14 08:23:54,777 INFO > > org.apache.flink.runtime.webmonitor.WebMonitorUtils - > Determined > > location of JobManager log file: > > > > > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > > 2016-04-14 08:23:54,777 INFO > > org.apache.flink.runtime.webmonitor.WebMonitorUtils - > Determined > > location of JobManager stdout file: > > > > > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out > > 2016-04-14 08:23:54,805 INFO > > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using > > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the web > > interface files > > 2016-04-14 08:23:54,805 INFO > > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using > > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 for > > web frontend JAR file uploads > > 2016-04-14 08:23:55,600 INFO > > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web > > frontend listening at 0:0:0:0:0:0:0:0:8081 > > 2016-04-14 08:23:55,601 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManager actor > > 2016-04-14 08:23:55,619 INFO org.apache.flink.runtime.blob.BlobServer > > - Created BLOB server storage directory > > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 > > 2016-04-14 08:23:55,634 INFO org.apache.flink.runtime.blob.BlobServer > > - Started BLOB server at 0.0.0.0:60439 - max concurrent > > requests: 50 - max backlog: 1000 > > 2016-04-14 08:23:55,653 INFO > > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - Using > job > > manager savepoint state backend. > > 2016-04-14 08:23:55,678 INFO > > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting > > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager on > > port > > 8081 > > 2016-04-14 08:23:55,678 INFO > > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New > leader > > reachable under akka.tcp://flink@172.31.45.232:6123/user/jobmanager:null > . > > 2016-04-14 08:23:55,692 INFO > > org.apache.flink.runtime.jobmanager.JobManager - Starting > > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. > > 2016-04-14 08:23:55,696 INFO > > org.apache.flink.runtime.jobmanager.MemoryArchivist - Started > > memory archivist akka://flink/user/archive > > 2016-04-14 08:23:55,702 INFO > > org.apache.flink.runtime.jobmanager.JobManager - > JobManager > > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted > leadership > > with leader session ID None. > > 2016-04-14 08:24:12,740 INFO > > org.apache.flink.runtime.instance.InstanceManager - > Registered > > TaskManager at tresor-testflinkth (akka.tcp:// > > flink@172.31.45.130:42189/user/taskmanager) as > > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts is > 1. > > Current number of alive task slots is 1. > > 2016-04-14 08:24:17,855 INFO > > org.apache.flink.runtime.instance.InstanceManager - > Registered > > TaskManager at tresor-testflinkth (akka.tcp:// > > flink@172.31.34.121:58814/user/taskmanager) as > > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts is > 2. > > Current number of alive task slots is 2. > > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter > > - dropping message [class > > akka.actor.ActorSelectionMessage] for non-local recipient > > [Actor[akka.tcp:// > > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// > > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// > > flink@172.31.45.232:6123] > > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor > > - Association with remote system [akka.tcp:// > > flink@127.0.0.1:35953] has failed, address is now gated for [5000] ms. > > Reason is: [Disassociated]. > > > > > > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the public > > address of the VM hosting job manager and 172.31.45.232 > > <http://flink@172.31.45.232:6123> is it private address. > > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> and > > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> are > the > > private address of the task manager > > > > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > I'm referring to the jobmanager.log file not the client log file. You > can > > > find it in the `/log` directory. > > > > > > Cheers, > > > Till > > > > > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> wrote: > > > > > > > Hi Till > > > > > > > > Thanks for the prompt reply. > > > > > > > > The logs say that Please make sure that the actor is running and its > > port > > > > is reachable. > > > > And it is actaully reachable because I can ping that address. > > > > > > > > Ned. > > > > > > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < > > [hidden email]> > > > > wrote: > > > > > > > > > Hi Ned, > > > > > > > > > > what does the logs of the JobManager say? > > > > > > > > > > Cheers, > > > > > Till > > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: > > > > > > > > > > > Hi everybody, > > > > > > > > > > > > I'm Ned, a young and passionte developer of apache technologies. > I > > > have > > > > > > been playing with apache flink lastly. > > > > > > > > > > > > This is what I wanted to do submit a flink topology to a remote > > flink > > > > > > cluster. The following are the steps that I did. > > > > > > > > > > > > - Install flink as a cluster indicated on the link > > > > > > > > > > > > > > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > > > > > > on three remotes VMs. > > > > > > - Run the sample WordCountRemoteByClient > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > > > > > > > > > > > > > by > > > > > > changing > > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to > > > > > > conf.put(Config.NIMBUS_HOST, > > > "publicIpOfJobmanagerInMyRemoteCluster"); > > > > > > > > > > > > Unfortunately for me when I run that program, I have a the > > following > > > > > > exception. > > > > > > > > > > > > org.apache.flink.client.program.ProgramInvocationException: The > > main > > > > > method > > > > > > caused an error. > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > > > > > > at > > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > > > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > > > > > > at > org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > > > > > > Caused by: java.lang.RuntimeException: Could not connect to Flink > > > > > > JobManager with address > publicIpOfJobmanagerInMyRemoteCluster:6123 > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > > at java.lang.reflect.Method.invoke(Method.java:483) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > > > > > > ... 6 more > > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not > > > > reachable. > > > > > > Please make sure that the actor is running and its port is > > reachable. > > > > > > at > > > > > > > > > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > > > > > > ... 14 more > > > > > > Caused by: java.util.concurrent.TimeoutException: Futures timed > out > > > > after > > > > > > [10000 milliseconds] > > > > > > at > > > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > > > > at > > > > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > > > > at scala.concurrent.Await$.result(package.scala:107) > > > > > > at > > > > > > > > > > > > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > > > > > > ... 19 more > > > > > > > > > > > > I try ping my jobmanager with > > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the > following > > > as > > > > > > responces. > > > > > > > > > > > > curl: (52) Empty reply from server > > > > > > > > > > > > Which is an indication that the job manager is reachable. > > > > > > > > > > > > So I was wondering if I doing it the right way. Please any help > > will > > > be > > > > > > welcoming. > > > > > > > > > > > > Thanks, > > > > > > Ned > > > > > > > > > > > > > > > > > > > > > |
Here a try that I given. As first I was configuring my cluster with private
ip and it was starting properly. So to avoid this akka issue, I decided to configure my cluster with public address but with this configuration, my cluster is not starting at all. Here is the logs that I get 2016-04-14 09:08:52,845 INFO org.apache.flink.runtime.jobmanager.JobManager - -------------------------------------------------------------------------------- 2016-04-14 09:08:52,847 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) 2016-04-14 09:08:52,847 INFO org.apache.flink.runtime.jobmanager.JobManager - Current user: ubuntu 2016-04-14 09:08:52,847 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 2016-04-14 09:08:52,847 INFO org.apache.flink.runtime.jobmanager.JobManager - Maximum heap size: 247 MiBytes 2016-04-14 09:08:52,847 INFO org.apache.flink.runtime.jobmanager.JobManager - JAVA_HOME: (not set) 2016-04-14 09:08:52,880 INFO org.apache.flink.runtime.jobmanager.JobManager - Hadoop version: 1.2.1 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM Options: 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xms256m 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xmx256m 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - -XX:MaxPermSize=256m 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog.file=/home/ubuntu/flink/log/flink-ubuntu-jobmanager-1-tresor-thiswithpublicip.479319055.streamly.com.log 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - Program Arguments: 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - --configDir 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - /home/ubuntu/flink/conf 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - --executionMode 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - cluster 2016-04-14 09:08:52,881 INFO org.apache.flink.runtime.jobmanager.JobManager - Classpath: /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: 2016-04-14 09:08:52,882 INFO org.apache.flink.runtime.jobmanager.JobManager - -------------------------------------------------------------------------------- 2016-04-14 09:08:52,883 INFO org.apache.flink.runtime.jobmanager.JobManager - Registered UNIX signal handlers for [TERM, HUP, INT] 2016-04-14 09:08:53,324 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /home/ubuntu/flink/conf 2016-04-14 09:08:53,373 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager without high-availability 2016-04-14 09:08:53,390 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on 54.233.145.230:6123 with execution mode CLUSTER 2016-04-14 09:08:53,613 INFO org.apache.flink.runtime.jobmanager.JobManager - Security is not enabled. Starting non-authenticated JobManager. 2016-04-14 09:08:53,651 INFO org.apache.flink.util.NetUtils - Unable to allocate on port 6123, due to error: Cannot assign requested address 2016-04-14 09:08:53,654 ERROR org.apache.flink.runtime.jobmanager.JobManager - Failed to run JobManager. java.lang.RuntimeException: Unable to do further retries starting the actor system at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:1779) at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1719) at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1655) at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) On Thu, Apr 14, 2016 at 10:11 AM, ned dogg <[hidden email]> wrote: > Thanks Till for the reply. > > But according to you how can I address that? > > Thanks, > Ned > > On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann <[hidden email]> > wrote: > >> Hi Ned, >> >> I think you are facing the issue described in this JIRA issue [1]. The >> problem is that you have a private and a public IP address and that Akka >> binds to the private IP address. Since the registered IP of an ActorSystem >> and the target IP address of a request to this ActorSystem have to be >> matching, you cannot reach the ActorSystem via the public IP address. >> Requests with a non-matching IP address are discarded, as indicated by the >> last log statements. >> >> [1] https://issues.apache.org/jira/browse/FLINK-2821 >> >> Cheers, >> Till >> >> On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> wrote: >> >> > 2016-04-14 08:23:51,900 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > >> > >> -------------------------------------------------------------------------------- >> > 2016-04-14 08:23:51,902 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) >> > 2016-04-14 08:23:51,902 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Current >> > user: root >> > 2016-04-14 08:23:51,902 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - JVM: >> > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> > 2016-04-14 08:23:51,902 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Maximum >> > heap size: 247 MiBytes >> > 2016-04-14 08:23:51,902 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 >> > 2016-04-14 08:23:51,929 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - Hadoop >> > version: 1.2.1 >> > 2016-04-14 08:23:51,929 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - JVM >> > Options: >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > -Xms256m >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > -Xmx256m >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > -XX:MaxPermSize=256m >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > >> > >> -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Program >> > Arguments: >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > --configDir >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > /home/ubuntu/flink/conf >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > --executionMode >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > cluster >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > Classpath: >> > >> > >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> > 2016-04-14 08:23:51,930 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> > >> > >> -------------------------------------------------------------------------------- >> > 2016-04-14 08:23:51,931 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Registered >> > UNIX signal handlers for [TERM, HUP, INT] >> > 2016-04-14 08:23:52,362 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - Loading >> > configuration from /home/ubuntu/flink/conf >> > 2016-04-14 08:23:52,400 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager without high-availability >> > 2016-04-14 08:23:52,408 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager on 172.31.45.232:6123 with execution mode CLUSTER >> > 2016-04-14 08:23:52,655 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Security >> > is not enabled. Starting non-authenticated JobManager. >> > 2016-04-14 08:23:52,701 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager >> > 2016-04-14 08:23:52,701 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager actor system at 172.31.45.232:6123 >> > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger >> > - Slf4jLogger started >> > 2016-04-14 08:23:54,293 INFO Remoting >> > - Starting remoting >> > 2016-04-14 08:23:54,712 INFO Remoting >> > - Remoting started; listening on addresses >> :[akka.tcp:// >> > flink@172.31.45.232:6123] >> > 2016-04-14 08:23:54,732 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManger web frontend >> > 2016-04-14 08:23:54,777 INFO >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> Determined >> > location of JobManager log file: >> > >> > >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> > 2016-04-14 08:23:54,777 INFO >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> Determined >> > location of JobManager stdout file: >> > >> > >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out >> > 2016-04-14 08:23:54,805 INFO >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using >> > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the >> web >> > interface files >> > 2016-04-14 08:23:54,805 INFO >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using >> > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 for >> > web frontend JAR file uploads >> > 2016-04-14 08:23:55,600 INFO >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web >> > frontend listening at 0:0:0:0:0:0:0:0:8081 >> > 2016-04-14 08:23:55,601 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager actor >> > 2016-04-14 08:23:55,619 INFO org.apache.flink.runtime.blob.BlobServer >> > - Created BLOB server storage directory >> > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 >> > 2016-04-14 08:23:55,634 INFO org.apache.flink.runtime.blob.BlobServer >> > - Started BLOB server at 0.0.0.0:60439 - max >> concurrent >> > requests: 50 - max backlog: 1000 >> > 2016-04-14 08:23:55,653 INFO >> > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - Using >> job >> > manager savepoint state backend. >> > 2016-04-14 08:23:55,678 INFO >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Starting >> > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager on >> > port >> > 8081 >> > 2016-04-14 08:23:55,678 INFO >> > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New >> leader >> > reachable under akka.tcp:// >> flink@172.31.45.232:6123/user/jobmanager:null. >> > 2016-04-14 08:23:55,692 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> Starting >> > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. >> > 2016-04-14 08:23:55,696 INFO >> > org.apache.flink.runtime.jobmanager.MemoryArchivist - Started >> > memory archivist akka://flink/user/archive >> > 2016-04-14 08:23:55,702 INFO >> > org.apache.flink.runtime.jobmanager.JobManager - >> JobManager >> > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted >> leadership >> > with leader session ID None. >> > 2016-04-14 08:24:12,740 INFO >> > org.apache.flink.runtime.instance.InstanceManager - >> Registered >> > TaskManager at tresor-testflinkth (akka.tcp:// >> > flink@172.31.45.130:42189/user/taskmanager) as >> > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts is >> 1. >> > Current number of alive task slots is 1. >> > 2016-04-14 08:24:17,855 INFO >> > org.apache.flink.runtime.instance.InstanceManager - >> Registered >> > TaskManager at tresor-testflinkth (akka.tcp:// >> > flink@172.31.34.121:58814/user/taskmanager) as >> > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts is >> 2. >> > Current number of alive task slots is 2. >> > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter >> > - dropping message [class >> > akka.actor.ActorSelectionMessage] for non-local recipient >> > [Actor[akka.tcp:// >> > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// >> > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// >> > flink@172.31.45.232:6123] >> > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor >> > - Association with remote system [akka.tcp:// >> > flink@127.0.0.1:35953] has failed, address is now gated for [5000] ms. >> > Reason is: [Disassociated]. >> > >> > >> > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the public >> > address of the VM hosting job manager and 172.31.45.232 >> > <http://flink@172.31.45.232:6123> is it private address. >> > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> and >> > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> are >> the >> > private address of the task manager >> > >> > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email]> >> > wrote: >> > >> > > I'm referring to the jobmanager.log file not the client log file. You >> can >> > > find it in the `/log` directory. >> > > >> > > Cheers, >> > > Till >> > > >> > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> >> wrote: >> > > >> > > > Hi Till >> > > > >> > > > Thanks for the prompt reply. >> > > > >> > > > The logs say that Please make sure that the actor is running and its >> > port >> > > > is reachable. >> > > > And it is actaully reachable because I can ping that address. >> > > > >> > > > Ned. >> > > > >> > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < >> > [hidden email]> >> > > > wrote: >> > > > >> > > > > Hi Ned, >> > > > > >> > > > > what does the logs of the JobManager say? >> > > > > >> > > > > Cheers, >> > > > > Till >> > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> wrote: >> > > > > >> > > > > > Hi everybody, >> > > > > > >> > > > > > I'm Ned, a young and passionte developer of apache >> technologies. I >> > > have >> > > > > > been playing with apache flink lastly. >> > > > > > >> > > > > > This is what I wanted to do submit a flink topology to a remote >> > flink >> > > > > > cluster. The following are the steps that I did. >> > > > > > >> > > > > > - Install flink as a cluster indicated on the link >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html >> > > > > > on three remotes VMs. >> > > > > > - Run the sample WordCountRemoteByClient >> > > > > > < >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java >> > > > > > > >> > > > > > by >> > > > > > changing >> > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to >> > > > > > conf.put(Config.NIMBUS_HOST, >> > > "publicIpOfJobmanagerInMyRemoteCluster"); >> > > > > > >> > > > > > Unfortunately for me when I run that program, I have a the >> > following >> > > > > > exception. >> > > > > > >> > > > > > org.apache.flink.client.program.ProgramInvocationException: The >> > main >> > > > > method >> > > > > > caused an error. >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) >> > > > > > at >> > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) >> > > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) >> > > > > > at >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) >> > > > > > Caused by: java.lang.RuntimeException: Could not connect to >> Flink >> > > > > > JobManager with address >> publicIpOfJobmanagerInMyRemoteCluster:6123 >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > > > > > at java.lang.reflect.Method.invoke(Method.java:483) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) >> > > > > > ... 6 more >> > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ >> > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not >> > > > reachable. >> > > > > > Please make sure that the actor is running and its port is >> > reachable. >> > > > > > at >> > > > > > >> > > > >> > >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) >> > > > > > ... 14 more >> > > > > > Caused by: java.util.concurrent.TimeoutException: Futures timed >> out >> > > > after >> > > > > > [10000 milliseconds] >> > > > > > at >> > > > >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> > > > > > at >> > > > >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> > > > > > at >> > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> > > > > > at scala.concurrent.Await$.result(package.scala:107) >> > > > > > at >> > > > > > >> > > > >> > >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) >> > > > > > ... 19 more >> > > > > > >> > > > > > I try ping my jobmanager with >> > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the >> following >> > > as >> > > > > > responces. >> > > > > > >> > > > > > curl: (52) Empty reply from server >> > > > > > >> > > > > > Which is an indication that the job manager is reachable. >> > > > > > >> > > > > > So I was wondering if I doing it the right way. Please any help >> > will >> > > be >> > > > > > welcoming. >> > > > > > >> > > > > > Thanks, >> > > > > > Ned >> > > > > > >> > > > > >> > > > >> > > >> > >> > > |
The log says: Unable to allocate on port 6123, due to error: Cannot assign
requested address Thus, I would assume that something with your cluster configuration is not entirely correct. Could you check that? On Thu, Apr 14, 2016 at 11:19 AM, ned dogg <[hidden email]> wrote: > Here a try that I given. As first I was configuring my cluster with private > ip and it was starting properly. > So to avoid this akka issue, I decided to configure my cluster with public > address but with this configuration, my cluster is not starting at all. > Here is the logs that I get > 2016-04-14 09:08:52,845 INFO > org.apache.flink.runtime.jobmanager.JobManager - > > -------------------------------------------------------------------------------- > 2016-04-14 09:08:52,847 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) > 2016-04-14 09:08:52,847 INFO > org.apache.flink.runtime.jobmanager.JobManager - Current > user: ubuntu > 2016-04-14 09:08:52,847 INFO > org.apache.flink.runtime.jobmanager.JobManager - JVM: > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 > 2016-04-14 09:08:52,847 INFO > org.apache.flink.runtime.jobmanager.JobManager - Maximum > heap size: 247 MiBytes > 2016-04-14 09:08:52,847 INFO > org.apache.flink.runtime.jobmanager.JobManager - > JAVA_HOME: (not set) > 2016-04-14 09:08:52,880 INFO > org.apache.flink.runtime.jobmanager.JobManager - Hadoop > version: 1.2.1 > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - JVM > Options: > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Xms256m > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Xmx256m > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -XX:MaxPermSize=256m > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > > -Dlog.file=/home/ubuntu/flink/log/flink-ubuntu-jobmanager-1-tresor-thiswithpublicip.479319055.streamly.com.log > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - Program > Arguments: > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > --configDir > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > /home/ubuntu/flink/conf > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > --executionMode > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > cluster > 2016-04-14 09:08:52,881 INFO > org.apache.flink.runtime.jobmanager.JobManager - > Classpath: > > /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: > 2016-04-14 09:08:52,882 INFO > org.apache.flink.runtime.jobmanager.JobManager - > > -------------------------------------------------------------------------------- > 2016-04-14 09:08:52,883 INFO > org.apache.flink.runtime.jobmanager.JobManager - Registered > UNIX signal handlers for [TERM, HUP, INT] > 2016-04-14 09:08:53,324 INFO > org.apache.flink.runtime.jobmanager.JobManager - Loading > configuration from /home/ubuntu/flink/conf > 2016-04-14 09:08:53,373 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager without high-availability > 2016-04-14 09:08:53,390 INFO > org.apache.flink.runtime.jobmanager.JobManager - Starting > JobManager on 54.233.145.230:6123 with execution mode CLUSTER > 2016-04-14 09:08:53,613 INFO > org.apache.flink.runtime.jobmanager.JobManager - Security > is not enabled. Starting non-authenticated JobManager. > 2016-04-14 09:08:53,651 INFO org.apache.flink.util.NetUtils > - Unable to allocate on port 6123, due to error: Cannot > assign requested address > 2016-04-14 09:08:53,654 ERROR > org.apache.flink.runtime.jobmanager.JobManager - Failed to > run JobManager. > java.lang.RuntimeException: Unable to do further retries starting the actor > system > at > > org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:1779) > at > > org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1719) > at > org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1655) > at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) > > > > On Thu, Apr 14, 2016 at 10:11 AM, ned dogg <[hidden email]> wrote: > > > Thanks Till for the reply. > > > > But according to you how can I address that? > > > > Thanks, > > Ned > > > > On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann <[hidden email]> > > wrote: > > > >> Hi Ned, > >> > >> I think you are facing the issue described in this JIRA issue [1]. The > >> problem is that you have a private and a public IP address and that Akka > >> binds to the private IP address. Since the registered IP of an > ActorSystem > >> and the target IP address of a request to this ActorSystem have to be > >> matching, you cannot reach the ActorSystem via the public IP address. > >> Requests with a non-matching IP address are discarded, as indicated by > the > >> last log statements. > >> > >> [1] https://issues.apache.org/jira/browse/FLINK-2821 > >> > >> Cheers, > >> Till > >> > >> On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> wrote: > >> > >> > 2016-04-14 08:23:51,900 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > > >> > > >> > -------------------------------------------------------------------------------- > >> > 2016-04-14 08:23:51,902 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 > UTC) > >> > 2016-04-14 08:23:51,902 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Current > >> > user: root > >> > 2016-04-14 08:23:51,902 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - JVM: > >> > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 > >> > 2016-04-14 08:23:51,902 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Maximum > >> > heap size: 247 MiBytes > >> > 2016-04-14 08:23:51,902 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 > >> > 2016-04-14 08:23:51,929 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > Hadoop > >> > version: 1.2.1 > >> > 2016-04-14 08:23:51,929 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - JVM > >> > Options: > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > -Xms256m > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > -Xmx256m > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > -XX:MaxPermSize=256m > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > > >> > > >> > -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Program > >> > Arguments: > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > --configDir > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > /home/ubuntu/flink/conf > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > --executionMode > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > cluster > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > Classpath: > >> > > >> > > >> > /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: > >> > 2016-04-14 08:23:51,930 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> > > >> > > >> > -------------------------------------------------------------------------------- > >> > 2016-04-14 08:23:51,931 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Registered > >> > UNIX signal handlers for [TERM, HUP, INT] > >> > 2016-04-14 08:23:52,362 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > Loading > >> > configuration from /home/ubuntu/flink/conf > >> > 2016-04-14 08:23:52,400 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager without high-availability > >> > 2016-04-14 08:23:52,408 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager on 172.31.45.232:6123 with execution mode CLUSTER > >> > 2016-04-14 08:23:52,655 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Security > >> > is not enabled. Starting non-authenticated JobManager. > >> > 2016-04-14 08:23:52,701 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager > >> > 2016-04-14 08:23:52,701 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager actor system at 172.31.45.232:6123 > >> > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger > >> > - Slf4jLogger started > >> > 2016-04-14 08:23:54,293 INFO Remoting > >> > - Starting remoting > >> > 2016-04-14 08:23:54,712 INFO Remoting > >> > - Remoting started; listening on addresses > >> :[akka.tcp:// > >> > flink@172.31.45.232:6123] > >> > 2016-04-14 08:23:54,732 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManger web frontend > >> > 2016-04-14 08:23:54,777 INFO > >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - > >> Determined > >> > location of JobManager log file: > >> > > >> > > >> > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > >> > 2016-04-14 08:23:54,777 INFO > >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - > >> Determined > >> > location of JobManager stdout file: > >> > > >> > > >> > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out > >> > 2016-04-14 08:23:54,805 INFO > >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using > >> > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the > >> web > >> > interface files > >> > 2016-04-14 08:23:54,805 INFO > >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using > >> > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 > for > >> > web frontend JAR file uploads > >> > 2016-04-14 08:23:55,600 INFO > >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web > >> > frontend listening at 0:0:0:0:0:0:0:0:8081 > >> > 2016-04-14 08:23:55,601 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager actor > >> > 2016-04-14 08:23:55,619 INFO org.apache.flink.runtime.blob.BlobServer > >> > - Created BLOB server storage directory > >> > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 > >> > 2016-04-14 08:23:55,634 INFO org.apache.flink.runtime.blob.BlobServer > >> > - Started BLOB server at 0.0.0.0:60439 - max > >> concurrent > >> > requests: 50 - max backlog: 1000 > >> > 2016-04-14 08:23:55,653 INFO > >> > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - Using > >> job > >> > manager savepoint state backend. > >> > 2016-04-14 08:23:55,678 INFO > >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - > >> Starting > >> > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager > on > >> > port > >> > 8081 > >> > 2016-04-14 08:23:55,678 INFO > >> > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New > >> leader > >> > reachable under akka.tcp:// > >> flink@172.31.45.232:6123/user/jobmanager:null. > >> > 2016-04-14 08:23:55,692 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Starting > >> > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. > >> > 2016-04-14 08:23:55,696 INFO > >> > org.apache.flink.runtime.jobmanager.MemoryArchivist - > Started > >> > memory archivist akka://flink/user/archive > >> > 2016-04-14 08:23:55,702 INFO > >> > org.apache.flink.runtime.jobmanager.JobManager - > >> JobManager > >> > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted > >> leadership > >> > with leader session ID None. > >> > 2016-04-14 08:24:12,740 INFO > >> > org.apache.flink.runtime.instance.InstanceManager - > >> Registered > >> > TaskManager at tresor-testflinkth (akka.tcp:// > >> > flink@172.31.45.130:42189/user/taskmanager) as > >> > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts > is > >> 1. > >> > Current number of alive task slots is 1. > >> > 2016-04-14 08:24:17,855 INFO > >> > org.apache.flink.runtime.instance.InstanceManager - > >> Registered > >> > TaskManager at tresor-testflinkth (akka.tcp:// > >> > flink@172.31.34.121:58814/user/taskmanager) as > >> > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts > is > >> 2. > >> > Current number of alive task slots is 2. > >> > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter > >> > - dropping message [class > >> > akka.actor.ActorSelectionMessage] for non-local recipient > >> > [Actor[akka.tcp:// > >> > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// > >> > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// > >> > flink@172.31.45.232:6123] > >> > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor > >> > - Association with remote system [akka.tcp:// > >> > flink@127.0.0.1:35953] has failed, address is now gated for [5000] > ms. > >> > Reason is: [Disassociated]. > >> > > >> > > >> > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the public > >> > address of the VM hosting job manager and 172.31.45.232 > >> > <http://flink@172.31.45.232:6123> is it private address. > >> > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> and > >> > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> are > >> the > >> > private address of the task manager > >> > > >> > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email]> > >> > wrote: > >> > > >> > > I'm referring to the jobmanager.log file not the client log file. > You > >> can > >> > > find it in the `/log` directory. > >> > > > >> > > Cheers, > >> > > Till > >> > > > >> > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> > >> wrote: > >> > > > >> > > > Hi Till > >> > > > > >> > > > Thanks for the prompt reply. > >> > > > > >> > > > The logs say that Please make sure that the actor is running and > its > >> > port > >> > > > is reachable. > >> > > > And it is actaully reachable because I can ping that address. > >> > > > > >> > > > Ned. > >> > > > > >> > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < > >> > [hidden email]> > >> > > > wrote: > >> > > > > >> > > > > Hi Ned, > >> > > > > > >> > > > > what does the logs of the JobManager say? > >> > > > > > >> > > > > Cheers, > >> > > > > Till > >> > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> > wrote: > >> > > > > > >> > > > > > Hi everybody, > >> > > > > > > >> > > > > > I'm Ned, a young and passionte developer of apache > >> technologies. I > >> > > have > >> > > > > > been playing with apache flink lastly. > >> > > > > > > >> > > > > > This is what I wanted to do submit a flink topology to a > remote > >> > flink > >> > > > > > cluster. The following are the steps that I did. > >> > > > > > > >> > > > > > - Install flink as a cluster indicated on the link > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > >> > > > > > on three remotes VMs. > >> > > > > > - Run the sample WordCountRemoteByClient > >> > > > > > < > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > >> > > > > > > > >> > > > > > by > >> > > > > > changing > >> > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to > >> > > > > > conf.put(Config.NIMBUS_HOST, > >> > > "publicIpOfJobmanagerInMyRemoteCluster"); > >> > > > > > > >> > > > > > Unfortunately for me when I run that program, I have a the > >> > following > >> > > > > > exception. > >> > > > > > > >> > > > > > org.apache.flink.client.program.ProgramInvocationException: > The > >> > main > >> > > > > method > >> > > > > > caused an error. > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > >> > > > > > at > >> > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > >> > > > > > at > org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > >> > > > > > at > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > >> > > > > > at > >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > >> > > > > > Caused by: java.lang.RuntimeException: Could not connect to > >> Flink > >> > > > > > JobManager with address > >> publicIpOfJobmanagerInMyRemoteCluster:6123 > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > >> > > > > > at > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > > > > > at java.lang.reflect.Method.invoke(Method.java:483) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > >> > > > > > ... 6 more > >> > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > >> > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager not > >> > > > reachable. > >> > > > > > Please make sure that the actor is running and its port is > >> > reachable. > >> > > > > > at > >> > > > > > > >> > > > > >> > > >> > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > >> > > > > > at > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > >> > > > > > ... 14 more > >> > > > > > Caused by: java.util.concurrent.TimeoutException: Futures > timed > >> out > >> > > > after > >> > > > > > [10000 milliseconds] > >> > > > > > at > >> > > > > >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > >> > > > > > at > >> > > > > >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > >> > > > > > at > >> > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > >> > > > > > at > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > >> > > > > > at scala.concurrent.Await$.result(package.scala:107) > >> > > > > > at > >> > > > > > > >> > > > > >> > > >> > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > >> > > > > > ... 19 more > >> > > > > > > >> > > > > > I try ping my jobmanager with > >> > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the > >> following > >> > > as > >> > > > > > responces. > >> > > > > > > >> > > > > > curl: (52) Empty reply from server > >> > > > > > > >> > > > > > Which is an indication that the job manager is reachable. > >> > > > > > > >> > > > > > So I was wondering if I doing it the right way. Please any > help > >> > will > >> > > be > >> > > > > > welcoming. > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Ned > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > > |
Hi Ned,
what you also could do is to ssh to your remote cluster and submit the job using the private IP address which is reachable from within your cluster. I don't know whether that would be applicable to your use case. Cheers, Till On Fri, Apr 15, 2016 at 9:22 AM, Till Rohrmann <[hidden email]> wrote: > The log says: Unable to allocate on port 6123, due to error: Cannot assign > requested address > > Thus, I would assume that something with your cluster configuration is not > entirely correct. Could you check that? > > On Thu, Apr 14, 2016 at 11:19 AM, ned dogg <[hidden email]> wrote: > >> Here a try that I given. As first I was configuring my cluster with >> private >> ip and it was starting properly. >> So to avoid this akka issue, I decided to configure my cluster with public >> address but with this configuration, my cluster is not starting at all. >> Here is the logs that I get >> 2016-04-14 09:08:52,845 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -------------------------------------------------------------------------------- >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Current >> user: ubuntu >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - JVM: >> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Maximum >> heap size: 247 MiBytes >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> JAVA_HOME: (not set) >> 2016-04-14 09:08:52,880 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Hadoop >> version: 1.2.1 >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - JVM >> Options: >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Xms256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Xmx256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -XX:MaxPermSize=256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -Dlog.file=/home/ubuntu/flink/log/flink-ubuntu-jobmanager-1-tresor-thiswithpublicip.479319055.streamly.com.log >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Program >> Arguments: >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> --configDir >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> /home/ubuntu/flink/conf >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> --executionMode >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> cluster >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> Classpath: >> >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> 2016-04-14 09:08:52,882 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -------------------------------------------------------------------------------- >> 2016-04-14 09:08:52,883 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> Registered >> UNIX signal handlers for [TERM, HUP, INT] >> 2016-04-14 09:08:53,324 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Loading >> configuration from /home/ubuntu/flink/conf >> 2016-04-14 09:08:53,373 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager without high-availability >> 2016-04-14 09:08:53,390 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager on 54.233.145.230:6123 with execution mode CLUSTER >> 2016-04-14 09:08:53,613 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Security >> is not enabled. Starting non-authenticated JobManager. >> 2016-04-14 09:08:53,651 INFO org.apache.flink.util.NetUtils >> - Unable to allocate on port 6123, due to error: Cannot >> assign requested address >> 2016-04-14 09:08:53,654 ERROR >> org.apache.flink.runtime.jobmanager.JobManager - Failed to >> run JobManager. >> java.lang.RuntimeException: Unable to do further retries starting the >> actor >> system >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:1779) >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1719) >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1655) >> at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) >> >> >> >> On Thu, Apr 14, 2016 at 10:11 AM, ned dogg <[hidden email]> wrote: >> >> > Thanks Till for the reply. >> > >> > But according to you how can I address that? >> > >> > Thanks, >> > Ned >> > >> > On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann <[hidden email] >> > >> > wrote: >> > >> >> Hi Ned, >> >> >> >> I think you are facing the issue described in this JIRA issue [1]. The >> >> problem is that you have a private and a public IP address and that >> Akka >> >> binds to the private IP address. Since the registered IP of an >> ActorSystem >> >> and the target IP address of a request to this ActorSystem have to be >> >> matching, you cannot reach the ActorSystem via the public IP address. >> >> Requests with a non-matching IP address are discarded, as indicated by >> the >> >> last log statements. >> >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-2821 >> >> >> >> Cheers, >> >> Till >> >> >> >> On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> >> wrote: >> >> >> >> > 2016-04-14 08:23:51,900 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -------------------------------------------------------------------------------- >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 >> UTC) >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Current >> >> > user: root >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> JVM: >> >> > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Maximum >> >> > heap size: 247 MiBytes >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 >> >> > 2016-04-14 08:23:51,929 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> Hadoop >> >> > version: 1.2.1 >> >> > 2016-04-14 08:23:51,929 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - JVM >> >> > Options: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Xms256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Xmx256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -XX:MaxPermSize=256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Program >> >> > Arguments: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > --configDir >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > /home/ubuntu/flink/conf >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > --executionMode >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > cluster >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > Classpath: >> >> > >> >> > >> >> >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -------------------------------------------------------------------------------- >> >> > 2016-04-14 08:23:51,931 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Registered >> >> > UNIX signal handlers for [TERM, HUP, INT] >> >> > 2016-04-14 08:23:52,362 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> Loading >> >> > configuration from /home/ubuntu/flink/conf >> >> > 2016-04-14 08:23:52,400 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager without high-availability >> >> > 2016-04-14 08:23:52,408 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager on 172.31.45.232:6123 with execution mode CLUSTER >> >> > 2016-04-14 08:23:52,655 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Security >> >> > is not enabled. Starting non-authenticated JobManager. >> >> > 2016-04-14 08:23:52,701 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager >> >> > 2016-04-14 08:23:52,701 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager actor system at 172.31.45.232:6123 >> >> > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger >> >> > - Slf4jLogger started >> >> > 2016-04-14 08:23:54,293 INFO Remoting >> >> > - Starting remoting >> >> > 2016-04-14 08:23:54,712 INFO Remoting >> >> > - Remoting started; listening on addresses >> >> :[akka.tcp:// >> >> > flink@172.31.45.232:6123] >> >> > 2016-04-14 08:23:54,732 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManger web frontend >> >> > 2016-04-14 08:23:54,777 INFO >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> >> Determined >> >> > location of JobManager log file: >> >> > >> >> > >> >> >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> >> > 2016-04-14 08:23:54,777 INFO >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> >> Determined >> >> > location of JobManager stdout file: >> >> > >> >> > >> >> >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out >> >> > 2016-04-14 08:23:54,805 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Using >> >> > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the >> >> web >> >> > interface files >> >> > 2016-04-14 08:23:54,805 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Using >> >> > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 >> for >> >> > web frontend JAR file uploads >> >> > 2016-04-14 08:23:55,600 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web >> >> > frontend listening at 0:0:0:0:0:0:0:0:8081 >> >> > 2016-04-14 08:23:55,601 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager actor >> >> > 2016-04-14 08:23:55,619 INFO >> org.apache.flink.runtime.blob.BlobServer >> >> > - Created BLOB server storage directory >> >> > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 >> >> > 2016-04-14 08:23:55,634 INFO >> org.apache.flink.runtime.blob.BlobServer >> >> > - Started BLOB server at 0.0.0.0:60439 - max >> >> concurrent >> >> > requests: 50 - max backlog: 1000 >> >> > 2016-04-14 08:23:55,653 INFO >> >> > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - >> Using >> >> job >> >> > manager savepoint state backend. >> >> > 2016-04-14 08:23:55,678 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> >> Starting >> >> > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager >> on >> >> > port >> >> > 8081 >> >> > 2016-04-14 08:23:55,678 INFO >> >> > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New >> >> leader >> >> > reachable under akka.tcp:// >> >> flink@172.31.45.232:6123/user/jobmanager:null. >> >> > 2016-04-14 08:23:55,692 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. >> >> > 2016-04-14 08:23:55,696 INFO >> >> > org.apache.flink.runtime.jobmanager.MemoryArchivist - >> Started >> >> > memory archivist akka://flink/user/archive >> >> > 2016-04-14 08:23:55,702 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> JobManager >> >> > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted >> >> leadership >> >> > with leader session ID None. >> >> > 2016-04-14 08:24:12,740 INFO >> >> > org.apache.flink.runtime.instance.InstanceManager - >> >> Registered >> >> > TaskManager at tresor-testflinkth (akka.tcp:// >> >> > flink@172.31.45.130:42189/user/taskmanager) as >> >> > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts >> is >> >> 1. >> >> > Current number of alive task slots is 1. >> >> > 2016-04-14 08:24:17,855 INFO >> >> > org.apache.flink.runtime.instance.InstanceManager - >> >> Registered >> >> > TaskManager at tresor-testflinkth (akka.tcp:// >> >> > flink@172.31.34.121:58814/user/taskmanager) as >> >> > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts >> is >> >> 2. >> >> > Current number of alive task slots is 2. >> >> > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter >> >> > - dropping message [class >> >> > akka.actor.ActorSelectionMessage] for non-local recipient >> >> > [Actor[akka.tcp:// >> >> > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// >> >> > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// >> >> > flink@172.31.45.232:6123] >> >> > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor >> >> > - Association with remote system [akka.tcp:// >> >> > flink@127.0.0.1:35953] has failed, address is now gated for [5000] >> ms. >> >> > Reason is: [Disassociated]. >> >> > >> >> > >> >> > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the >> public >> >> > address of the VM hosting job manager and 172.31.45.232 >> >> > <http://flink@172.31.45.232:6123> is it private address. >> >> > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> >> and >> >> > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> >> are >> >> the >> >> > private address of the task manager >> >> > >> >> > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email] >> > >> >> > wrote: >> >> > >> >> > > I'm referring to the jobmanager.log file not the client log file. >> You >> >> can >> >> > > find it in the `/log` directory. >> >> > > >> >> > > Cheers, >> >> > > Till >> >> > > >> >> > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> >> >> wrote: >> >> > > >> >> > > > Hi Till >> >> > > > >> >> > > > Thanks for the prompt reply. >> >> > > > >> >> > > > The logs say that Please make sure that the actor is running and >> its >> >> > port >> >> > > > is reachable. >> >> > > > And it is actaully reachable because I can ping that address. >> >> > > > >> >> > > > Ned. >> >> > > > >> >> > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < >> >> > [hidden email]> >> >> > > > wrote: >> >> > > > >> >> > > > > Hi Ned, >> >> > > > > >> >> > > > > what does the logs of the JobManager say? >> >> > > > > >> >> > > > > Cheers, >> >> > > > > Till >> >> > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> >> wrote: >> >> > > > > >> >> > > > > > Hi everybody, >> >> > > > > > >> >> > > > > > I'm Ned, a young and passionte developer of apache >> >> technologies. I >> >> > > have >> >> > > > > > been playing with apache flink lastly. >> >> > > > > > >> >> > > > > > This is what I wanted to do submit a flink topology to a >> remote >> >> > flink >> >> > > > > > cluster. The following are the steps that I did. >> >> > > > > > >> >> > > > > > - Install flink as a cluster indicated on the link >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html >> >> > > > > > on three remotes VMs. >> >> > > > > > - Run the sample WordCountRemoteByClient >> >> > > > > > < >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java >> >> > > > > > > >> >> > > > > > by >> >> > > > > > changing >> >> > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to >> >> > > > > > conf.put(Config.NIMBUS_HOST, >> >> > > "publicIpOfJobmanagerInMyRemoteCluster"); >> >> > > > > > >> >> > > > > > Unfortunately for me when I run that program, I have a the >> >> > following >> >> > > > > > exception. >> >> > > > > > >> >> > > > > > org.apache.flink.client.program.ProgramInvocationException: >> The >> >> > main >> >> > > > > method >> >> > > > > > caused an error. >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) >> >> > > > > > at >> >> > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) >> >> > > > > > at >> org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) >> >> > > > > > at >> >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) >> >> > > > > > Caused by: java.lang.RuntimeException: Could not connect to >> >> Flink >> >> > > > > > JobManager with address >> >> publicIpOfJobmanagerInMyRemoteCluster:6123 >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) >> >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> > > > > > at java.lang.reflect.Method.invoke(Method.java:483) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) >> >> > > > > > ... 6 more >> >> > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ >> >> > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager >> not >> >> > > > reachable. >> >> > > > > > Please make sure that the actor is running and its port is >> >> > reachable. >> >> > > > > > at >> >> > > > > > >> >> > > > >> >> > >> >> >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) >> >> > > > > > ... 14 more >> >> > > > > > Caused by: java.util.concurrent.TimeoutException: Futures >> timed >> >> out >> >> > > > after >> >> > > > > > [10000 milliseconds] >> >> > > > > > at >> >> > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> >> > > > > > at >> >> > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> >> > > > > > at >> >> > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> >> > > > > > at scala.concurrent.Await$.result(package.scala:107) >> >> > > > > > at >> >> > > > > > >> >> > > > >> >> > >> >> >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) >> >> > > > > > ... 19 more >> >> > > > > > >> >> > > > > > I try ping my jobmanager with >> >> > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the >> >> following >> >> > > as >> >> > > > > > responces. >> >> > > > > > >> >> > > > > > curl: (52) Empty reply from server >> >> > > > > > >> >> > > > > > Which is an indication that the job manager is reachable. >> >> > > > > > >> >> > > > > > So I was wondering if I doing it the right way. Please any >> help >> >> > will >> >> > > be >> >> > > > > > welcoming. >> >> > > > > > >> >> > > > > > Thanks, >> >> > > > > > Ned >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >> > > |
Hi Till,
Thanks for the reply. The idea of ssh of the instance is a good one. I thought of that but in my case it is not applicable because I setting up a cluster some employees of a company. And ssh the instance by employees will mean giving them the instance's key-pair, which I can not. Matter of fact, I'm the one who is supposed to handle the maintainance of the cluster. Le Vendredi 15 avril 2016 10h34, Till Rohrmann <[hidden email]> a écrit : Hi Ned, what you also could do is to ssh to your remote cluster and submit the job using the private IP address which is reachable from within your cluster. I don't know whether that would be applicable to your use case. Cheers, Till On Fri, Apr 15, 2016 at 9:22 AM, Till Rohrmann <[hidden email]> wrote: > The log says: Unable to allocate on port 6123, due to error: Cannot assign > requested address > > Thus, I would assume that something with your cluster configuration is not > entirely correct. Could you check that? > > On Thu, Apr 14, 2016 at 11:19 AM, ned dogg <[hidden email]> wrote: > >> Here a try that I given. As first I was configuring my cluster with >> private >> ip and it was starting properly. >> So to avoid this akka issue, I decided to configure my cluster with public >> address but with this configuration, my cluster is not starting at all. >> Here is the logs that I get >> 2016-04-14 09:08:52,845 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -------------------------------------------------------------------------------- >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Current >> user: ubuntu >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - JVM: >> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Maximum >> heap size: 247 MiBytes >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> JAVA_HOME: (not set) >> 2016-04-14 09:08:52,880 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Hadoop >> version: 1.2.1 >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - JVM >> Options: >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Xms256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Xmx256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -XX:MaxPermSize=256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -Dlog.file=/home/ubuntu/flink/log/flink-ubuntu-jobmanager-1-tresor-thiswithpublicip.479319055.streamly.com.log >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Program >> Arguments: >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> --configDir >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> /home/ubuntu/flink/conf >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> --executionMode >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> cluster >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> Classpath: >> >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> 2016-04-14 09:08:52,882 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -------------------------------------------------------------------------------- >> 2016-04-14 09:08:52,883 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> Registered >> UNIX signal handlers for [TERM, HUP, INT] >> 2016-04-14 09:08:53,324 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Loading >> configuration from /home/ubuntu/flink/conf >> 2016-04-14 09:08:53,373 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager without high-availability >> 2016-04-14 09:08:53,390 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager on 54.233.145.230:6123 with execution mode CLUSTER >> 2016-04-14 09:08:53,613 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Security >> is not enabled. Starting non-authenticated JobManager. >> 2016-04-14 09:08:53,651 INFO org.apache.flink.util.NetUtils >> - Unable to allocate on port 6123, due to error: Cannot >> assign requested address >> 2016-04-14 09:08:53,654 ERROR >> org.apache.flink.runtime.jobmanager.JobManager - Failed to >> run JobManager. >> java.lang.RuntimeException: Unable to do further retries starting the >> actor >> system >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:1779) >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1719) >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1655) >> at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) >> >> >> >> On Thu, Apr 14, 2016 at 10:11 AM, ned dogg <[hidden email]> wrote: >> >> > Thanks Till for the reply. >> > >> > But according to you how can I address that? >> > >> > Thanks, >> > Ned >> > >> > On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann <[hidden email] >> > >> > wrote: >> > >> >> Hi Ned, >> >> >> >> I think you are facing the issue described in this JIRA issue [1]. The >> >> problem is that you have a private and a public IP address and that >> Akka >> >> binds to the private IP address. Since the registered IP of an >> ActorSystem >> >> and the target IP address of a request to this ActorSystem have to be >> >> matching, you cannot reach the ActorSystem via the public IP address. >> >> Requests with a non-matching IP address are discarded, as indicated by >> the >> >> last log statements. >> >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-2821 >> >> >> >> Cheers, >> >> Till >> >> >> >> On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> >> wrote: >> >> >> >> > 2016-04-14 08:23:51,900 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -------------------------------------------------------------------------------- >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 >> UTC) >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Current >> >> > user: root >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> JVM: >> >> > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Maximum >> >> > heap size: 247 MiBytes >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 >> >> > 2016-04-14 08:23:51,929 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> Hadoop >> >> > version: 1.2.1 >> >> > 2016-04-14 08:23:51,929 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - JVM >> >> > Options: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Xms256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Xmx256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -XX:MaxPermSize=256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Program >> >> > Arguments: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > --configDir >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > /home/ubuntu/flink/conf >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > --executionMode >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > cluster >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > Classpath: >> >> > >> >> > >> >> >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -------------------------------------------------------------------------------- >> >> > 2016-04-14 08:23:51,931 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Registered >> >> > UNIX signal handlers for [TERM, HUP, INT] >> >> > 2016-04-14 08:23:52,362 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> Loading >> >> > configuration from /home/ubuntu/flink/conf >> >> > 2016-04-14 08:23:52,400 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager without high-availability >> >> > 2016-04-14 08:23:52,408 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager on 172.31.45.232:6123 with execution mode CLUSTER >> >> > 2016-04-14 08:23:52,655 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Security >> >> > is not enabled. Starting non-authenticated JobManager. >> >> > 2016-04-14 08:23:52,701 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager >> >> > 2016-04-14 08:23:52,701 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager actor system at 172.31.45.232:6123 >> >> > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger >> >> > - Slf4jLogger started >> >> > 2016-04-14 08:23:54,293 INFO Remoting >> >> > - Starting remoting >> >> > 2016-04-14 08:23:54,712 INFO Remoting >> >> > - Remoting started; listening on addresses >> >> :[akka.tcp:// >> >> > flink@172.31.45.232:6123] >> >> > 2016-04-14 08:23:54,732 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManger web frontend >> >> > 2016-04-14 08:23:54,777 INFO >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> >> Determined >> >> > location of JobManager log file: >> >> > >> >> > >> >> >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> >> > 2016-04-14 08:23:54,777 INFO >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> >> Determined >> >> > location of JobManager stdout file: >> >> > >> >> > >> >> >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out >> >> > 2016-04-14 08:23:54,805 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Using >> >> > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the >> >> web >> >> > interface files >> >> > 2016-04-14 08:23:54,805 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Using >> >> > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 >> for >> >> > web frontend JAR file uploads >> >> > 2016-04-14 08:23:55,600 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web >> >> > frontend listening at 0:0:0:0:0:0:0:0:8081 >> >> > 2016-04-14 08:23:55,601 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager actor >> >> > 2016-04-14 08:23:55,619 INFO >> org.apache.flink.runtime.blob.BlobServer >> >> > - Created BLOB server storage directory >> >> > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 >> >> > 2016-04-14 08:23:55,634 INFO >> org.apache.flink.runtime.blob.BlobServer >> >> > - Started BLOB server at 0.0.0.0:60439 - max >> >> concurrent >> >> > requests: 50 - max backlog: 1000 >> >> > 2016-04-14 08:23:55,653 INFO >> >> > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - >> Using >> >> job >> >> > manager savepoint state backend. >> >> > 2016-04-14 08:23:55,678 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> >> Starting >> >> > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager >> on >> >> > port >> >> > 8081 >> >> > 2016-04-14 08:23:55,678 INFO >> >> > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New >> >> leader >> >> > reachable under akka.tcp:// >> >> flink@172.31.45.232:6123/user/jobmanager:null. >> >> > 2016-04-14 08:23:55,692 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. >> >> > 2016-04-14 08:23:55,696 INFO >> >> > org.apache.flink.runtime.jobmanager.MemoryArchivist - >> Started >> >> > memory archivist akka://flink/user/archive >> >> > 2016-04-14 08:23:55,702 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> JobManager >> >> > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted >> >> leadership >> >> > with leader session ID None. >> >> > 2016-04-14 08:24:12,740 INFO >> >> > org.apache.flink.runtime.instance.InstanceManager - >> >> Registered >> >> > TaskManager at tresor-testflinkth (akka.tcp:// >> >> > flink@172.31.45.130:42189/user/taskmanager) as >> >> > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts >> is >> >> 1. >> >> > Current number of alive task slots is 1. >> >> > 2016-04-14 08:24:17,855 INFO >> >> > org.apache.flink.runtime.instance.InstanceManager - >> >> Registered >> >> > TaskManager at tresor-testflinkth (akka.tcp:// >> >> > flink@172.31.34.121:58814/user/taskmanager) as >> >> > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts >> is >> >> 2. >> >> > Current number of alive task slots is 2. >> >> > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter >> >> > - dropping message [class >> >> > akka.actor.ActorSelectionMessage] for non-local recipient >> >> > [Actor[akka.tcp:// >> >> > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// >> >> > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// >> >> > flink@172.31.45.232:6123] >> >> > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor >> >> > - Association with remote system [akka.tcp:// >> >> > flink@127.0.0.1:35953] has failed, address is now gated for [5000] >> ms. >> >> > Reason is: [Disassociated]. >> >> > >> >> > >> >> > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the >> public >> >> > address of the VM hosting job manager and 172.31.45.232 >> >> > <http://flink@172.31.45.232:6123> is it private address. >> >> > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> >> and >> >> > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> >> are >> >> the >> >> > private address of the task manager >> >> > >> >> > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email] >> > >> >> > wrote: >> >> > >> >> > > I'm referring to the jobmanager.log file not the client log file. >> You >> >> can >> >> > > find it in the `/log` directory. >> >> > > >> >> > > Cheers, >> >> > > Till >> >> > > >> >> > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> >> >> wrote: >> >> > > >> >> > > > Hi Till >> >> > > > >> >> > > > Thanks for the prompt reply. >> >> > > > >> >> > > > The logs say that Please make sure that the actor is running and >> its >> >> > port >> >> > > > is reachable. >> >> > > > And it is actaully reachable because I can ping that address. >> >> > > > >> >> > > > Ned. >> >> > > > >> >> > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < >> >> > [hidden email]> >> >> > > > wrote: >> >> > > > >> >> > > > > Hi Ned, >> >> > > > > >> >> > > > > what does the logs of the JobManager say? >> >> > > > > >> >> > > > > Cheers, >> >> > > > > Till >> >> > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> >> wrote: >> >> > > > > >> >> > > > > > Hi everybody, >> >> > > > > > >> >> > > > > > I'm Ned, a young and passionte developer of apache >> >> technologies. I >> >> > > have >> >> > > > > > been playing with apache flink lastly. >> >> > > > > > >> >> > > > > > This is what I wanted to do submit a flink topology to a >> remote >> >> > flink >> >> > > > > > cluster. The following are the steps that I did. >> >> > > > > > >> >> > > > > > - Install flink as a cluster indicated on the link >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html >> >> > > > > > on three remotes VMs. >> >> > > > > > - Run the sample WordCountRemoteByClient >> >> > > > > > < >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java >> >> > > > > > > >> >> > > > > > by >> >> > > > > > changing >> >> > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to >> >> > > > > > conf.put(Config.NIMBUS_HOST, >> >> > > "publicIpOfJobmanagerInMyRemoteCluster"); >> >> > > > > > >> >> > > > > > Unfortunately for me when I run that program, I have a the >> >> > following >> >> > > > > > exception. >> >> > > > > > >> >> > > > > > org.apache.flink.client.program.ProgramInvocationException: >> The >> >> > main >> >> > > > > method >> >> > > > > > caused an error. >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) >> >> > > > > > at >> >> > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) >> >> > > > > > at >> org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) >> >> > > > > > at >> >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) >> >> > > > > > Caused by: java.lang.RuntimeException: Could not connect to >> >> Flink >> >> > > > > > JobManager with address >> >> publicIpOfJobmanagerInMyRemoteCluster:6123 >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) >> >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> > > > > > at java.lang.reflect.Method.invoke(Method.java:483) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) >> >> > > > > > ... 6 more >> >> > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ >> >> > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager >> not >> >> > > > reachable. >> >> > > > > > Please make sure that the actor is running and its port is >> >> > reachable. >> >> > > > > > at >> >> > > > > > >> >> > > > >> >> > >> >> >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) >> >> > > > > > ... 14 more >> >> > > > > > Caused by: java.util.concurrent.TimeoutException: Futures >> timed >> >> out >> >> > > > after >> >> > > > > > [10000 milliseconds] >> >> > > > > > at >> >> > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> >> > > > > > at >> >> > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> >> > > > > > at >> >> > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> >> > > > > > at scala.concurrent.Await$.result(package.scala:107) >> >> > > > > > at >> >> > > > > > >> >> > > > >> >> > >> >> >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) >> >> > > > > > ... 19 more >> >> > > > > > >> >> > > > > > I try ping my jobmanager with >> >> > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the >> >> following >> >> > > as >> >> > > > > > responces. >> >> > > > > > >> >> > > > > > curl: (52) Empty reply from server >> >> > > > > > >> >> > > > > > Which is an indication that the job manager is reachable. >> >> > > > > > >> >> > > > > > So I was wondering if I doing it the right way. Please any >> help >> >> > will >> >> > > be >> >> > > > > > welcoming. >> >> > > > > > >> >> > > > > > Thanks, >> >> > > > > > Ned >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >> > > |
In reply to this post by ned dogg
Hi Till/Ned,
Soory I thought this was my post. Le Vendredi 15 avril 2016 17h28, ned dogg <[hidden email]> a écrit : Hi Till/Jstar, Thanks for the the reply. Well I'm facing the same issue as Jstar. Here is my scenarios, I have app that is creating flink cluster on VMs for users. This app cannot create generate a file with the key-pair. So ssh the instance to deploy the flink topology wouldn't be possible as we can't provide them with key-pairs. On Fri, Apr 15, 2016 at 4:36 PM, star jlong <[hidden email]> wrote: Hi Till, Thanks for the reply. The idea of ssh of the instance is a good one. I thought of that but in my case it is not applicable because I setting up a cluster some employees of a company. And ssh the instance by employees will mean giving them the instance's key-pair, which I can not. Matter of fact, I'm the one who is supposed to handle the maintainance of the cluster. Le Vendredi 15 avril 2016 10h34, Till Rohrmann <[hidden email]> a écrit : Hi Ned, what you also could do is to ssh to your remote cluster and submit the job using the private IP address which is reachable from within your cluster. I don't know whether that would be applicable to your use case. Cheers, Till On Fri, Apr 15, 2016 at 9:22 AM, Till Rohrmann <[hidden email]> wrote: > The log says: Unable to allocate on port 6123, due to error: Cannot assign > requested address > > Thus, I would assume that something with your cluster configuration is not > entirely correct. Could you check that? > > On Thu, Apr 14, 2016 at 11:19 AM, ned dogg <[hidden email]> wrote: > >> Here a try that I given. As first I was configuring my cluster with >> private >> ip and it was starting properly. >> So to avoid this akka issue, I decided to configure my cluster with public >> address but with this configuration, my cluster is not starting at all. >> Here is the logs that I get >> 2016-04-14 09:08:52,845 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -------------------------------------------------------------------------------- >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Current >> user: ubuntu >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - JVM: >> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Maximum >> heap size: 247 MiBytes >> 2016-04-14 09:08:52,847 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> JAVA_HOME: (not set) >> 2016-04-14 09:08:52,880 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Hadoop >> version: 1.2.1 >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - JVM >> Options: >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Xms256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Xmx256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -XX:MaxPermSize=256m >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -Dlog.file=/home/ubuntu/flink/log/flink-ubuntu-jobmanager-1-tresor-thiswithpublicip.479319055.streamly.com.log >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Program >> Arguments: >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> --configDir >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> /home/ubuntu/flink/conf >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> --executionMode >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> cluster >> 2016-04-14 09:08:52,881 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> Classpath: >> >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> 2016-04-14 09:08:52,882 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> >> -------------------------------------------------------------------------------- >> 2016-04-14 09:08:52,883 INFO >> org.apache.flink.runtime.jobmanager.JobManager - >> Registered >> UNIX signal handlers for [TERM, HUP, INT] >> 2016-04-14 09:08:53,324 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Loading >> configuration from /home/ubuntu/flink/conf >> 2016-04-14 09:08:53,373 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager without high-availability >> 2016-04-14 09:08:53,390 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Starting >> JobManager on 54.233.145.230:6123 with execution mode CLUSTER >> 2016-04-14 09:08:53,613 INFO >> org.apache.flink.runtime.jobmanager.JobManager - Security >> is not enabled. Starting non-authenticated JobManager. >> 2016-04-14 09:08:53,651 INFO org.apache.flink.util.NetUtils >> - Unable to allocate on port 6123, due to error: Cannot >> assign requested address >> 2016-04-14 09:08:53,654 ERROR >> org.apache.flink.runtime.jobmanager.JobManager - Failed to >> run JobManager. >> java.lang.RuntimeException: Unable to do further retries starting the >> actor >> system >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:1779) >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1719) >> at >> >> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1655) >> at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) >> >> >> >> On Thu, Apr 14, 2016 at 10:11 AM, ned dogg <[hidden email]> wrote: >> >> > Thanks Till for the reply. >> > >> > But according to you how can I address that? >> > >> > Thanks, >> > Ned >> > >> > On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann <[hidden email] >> > >> > wrote: >> > >> >> Hi Ned, >> >> >> >> I think you are facing the issue described in this JIRA issue [1]. The >> >> problem is that you have a private and a public IP address and that >> Akka >> >> binds to the private IP address. Since the registered IP of an >> ActorSystem >> >> and the target IP address of a request to this ActorSystem have to be >> >> matching, you cannot reach the ActorSystem via the public IP address. >> >> Requests with a non-matching IP address are discarded, as indicated by >> the >> >> last log statements. >> >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-2821 >> >> >> >> Cheers, >> >> Till >> >> >> >> On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> >> wrote: >> >> >> >> > 2016-04-14 08:23:51,900 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -------------------------------------------------------------------------------- >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 >> UTC) >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Current >> >> > user: root >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> JVM: >> >> > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Maximum >> >> > heap size: 247 MiBytes >> >> > 2016-04-14 08:23:51,902 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 >> >> > 2016-04-14 08:23:51,929 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> Hadoop >> >> > version: 1.2.1 >> >> > 2016-04-14 08:23:51,929 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - JVM >> >> > Options: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Xms256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Xmx256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -XX:MaxPermSize=256m >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Program >> >> > Arguments: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > --configDir >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > /home/ubuntu/flink/conf >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > --executionMode >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > cluster >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > Classpath: >> >> > >> >> > >> >> >> /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: >> >> > 2016-04-14 08:23:51,930 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> > >> >> > >> >> >> -------------------------------------------------------------------------------- >> >> > 2016-04-14 08:23:51,931 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Registered >> >> > UNIX signal handlers for [TERM, HUP, INT] >> >> > 2016-04-14 08:23:52,362 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> Loading >> >> > configuration from /home/ubuntu/flink/conf >> >> > 2016-04-14 08:23:52,400 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager without high-availability >> >> > 2016-04-14 08:23:52,408 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager on 172.31.45.232:6123 with execution mode CLUSTER >> >> > 2016-04-14 08:23:52,655 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Security >> >> > is not enabled. Starting non-authenticated JobManager. >> >> > 2016-04-14 08:23:52,701 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager >> >> > 2016-04-14 08:23:52,701 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager actor system at 172.31.45.232:6123 >> >> > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger >> >> > - Slf4jLogger started >> >> > 2016-04-14 08:23:54,293 INFO Remoting >> >> > - Starting remoting >> >> > 2016-04-14 08:23:54,712 INFO Remoting >> >> > - Remoting started; listening on addresses >> >> :[akka.tcp:// >> >> > flink@172.31.45.232:6123] >> >> > 2016-04-14 08:23:54,732 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManger web frontend >> >> > 2016-04-14 08:23:54,777 INFO >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> >> Determined >> >> > location of JobManager log file: >> >> > >> >> > >> >> >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log >> >> > 2016-04-14 08:23:54,777 INFO >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - >> >> Determined >> >> > location of JobManager stdout file: >> >> > >> >> > >> >> >> /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out >> >> > 2016-04-14 08:23:54,805 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Using >> >> > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for the >> >> web >> >> > interface files >> >> > 2016-04-14 08:23:54,805 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> Using >> >> > directory /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 >> for >> >> > web frontend JAR file uploads >> >> > 2016-04-14 08:23:55,600 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web >> >> > frontend listening at 0:0:0:0:0:0:0:0:8081 >> >> > 2016-04-14 08:23:55,601 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager actor >> >> > 2016-04-14 08:23:55,619 INFO >> org.apache.flink.runtime.blob.BlobServer >> >> > - Created BLOB server storage directory >> >> > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 >> >> > 2016-04-14 08:23:55,634 INFO >> org.apache.flink.runtime.blob.BlobServer >> >> > - Started BLOB server at 0.0.0.0:60439 - max >> >> concurrent >> >> > requests: 50 - max backlog: 1000 >> >> > 2016-04-14 08:23:55,653 INFO >> >> > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - >> Using >> >> job >> >> > manager savepoint state backend. >> >> > 2016-04-14 08:23:55,678 INFO >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - >> >> Starting >> >> > with JobManager akka.tcp://flink@172.31.45.232:6123/user/jobmanager >> on >> >> > port >> >> > 8081 >> >> > 2016-04-14 08:23:55,678 INFO >> >> > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New >> >> leader >> >> > reachable under akka.tcp:// >> >> flink@172.31.45.232:6123/user/jobmanager:null. >> >> > 2016-04-14 08:23:55,692 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> Starting >> >> > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. >> >> > 2016-04-14 08:23:55,696 INFO >> >> > org.apache.flink.runtime.jobmanager.MemoryArchivist - >> Started >> >> > memory archivist akka://flink/user/archive >> >> > 2016-04-14 08:23:55,702 INFO >> >> > org.apache.flink.runtime.jobmanager.JobManager - >> >> JobManager >> >> > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted >> >> leadership >> >> > with leader session ID None. >> >> > 2016-04-14 08:24:12,740 INFO >> >> > org.apache.flink.runtime.instance.InstanceManager - >> >> Registered >> >> > TaskManager at tresor-testflinkth (akka.tcp:// >> >> > flink@172.31.45.130:42189/user/taskmanager) as >> >> > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered hosts >> is >> >> 1. >> >> > Current number of alive task slots is 1. >> >> > 2016-04-14 08:24:17,855 INFO >> >> > org.apache.flink.runtime.instance.InstanceManager - >> >> Registered >> >> > TaskManager at tresor-testflinkth (akka.tcp:// >> >> > flink@172.31.34.121:58814/user/taskmanager) as >> >> > fead50f6831aa3f341d58162bb918d90. Current number of registered hosts >> is >> >> 2. >> >> > Current number of alive task slots is 2. >> >> > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter >> >> > - dropping message [class >> >> > akka.actor.ActorSelectionMessage] for non-local recipient >> >> > [Actor[akka.tcp:// >> >> > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// >> >> > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// >> >> > flink@172.31.45.232:6123] >> >> > 2016-04-14 08:29:59,777 WARN akka.remote.ReliableDeliverySupervisor >> >> > - Association with remote system [akka.tcp:// >> >> > flink@127.0.0.1:35953] has failed, address is now gated for [5000] >> ms. >> >> > Reason is: [Disassociated]. >> >> > >> >> > >> >> > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the >> public >> >> > address of the VM hosting job manager and 172.31.45.232 >> >> > <http://flink@172.31.45.232:6123> is it private address. >> >> > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> >> and >> >> > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> >> are >> >> the >> >> > private address of the task manager >> >> > >> >> > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann <[hidden email] >> > >> >> > wrote: >> >> > >> >> > > I'm referring to the jobmanager.log file not the client log file. >> You >> >> can >> >> > > find it in the `/log` directory. >> >> > > >> >> > > Cheers, >> >> > > Till >> >> > > >> >> > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> >> >> wrote: >> >> > > >> >> > > > Hi Till >> >> > > > >> >> > > > Thanks for the prompt reply. >> >> > > > >> >> > > > The logs say that Please make sure that the actor is running and >> its >> >> > port >> >> > > > is reachable. >> >> > > > And it is actaully reachable because I can ping that address. >> >> > > > >> >> > > > Ned. >> >> > > > >> >> > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < >> >> > [hidden email]> >> >> > > > wrote: >> >> > > > >> >> > > > > Hi Ned, >> >> > > > > >> >> > > > > what does the logs of the JobManager say? >> >> > > > > >> >> > > > > Cheers, >> >> > > > > Till >> >> > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> >> wrote: >> >> > > > > >> >> > > > > > Hi everybody, >> >> > > > > > >> >> > > > > > I'm Ned, a young and passionte developer of apache >> >> technologies. I >> >> > > have >> >> > > > > > been playing with apache flink lastly. >> >> > > > > > >> >> > > > > > This is what I wanted to do submit a flink topology to a >> remote >> >> > flink >> >> > > > > > cluster. The following are the steps that I did. >> >> > > > > > >> >> > > > > > - Install flink as a cluster indicated on the link >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html >> >> > > > > > on three remotes VMs. >> >> > > > > > - Run the sample WordCountRemoteByClient >> >> > > > > > < >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java >> >> > > > > > > >> >> > > > > > by >> >> > > > > > changing >> >> > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to >> >> > > > > > conf.put(Config.NIMBUS_HOST, >> >> > > "publicIpOfJobmanagerInMyRemoteCluster"); >> >> > > > > > >> >> > > > > > Unfortunately for me when I run that program, I have a the >> >> > following >> >> > > > > > exception. >> >> > > > > > >> >> > > > > > org.apache.flink.client.program.ProgramInvocationException: >> The >> >> > main >> >> > > > > method >> >> > > > > > caused an error. >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) >> >> > > > > > at >> >> > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) >> >> > > > > > at >> org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) >> >> > > > > > at >> >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) >> >> > > > > > Caused by: java.lang.RuntimeException: Could not connect to >> >> Flink >> >> > > > > > JobManager with address >> >> publicIpOfJobmanagerInMyRemoteCluster:6123 >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) >> >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> > > > > > at java.lang.reflect.Method.invoke(Method.java:483) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) >> >> > > > > > ... 6 more >> >> > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ >> >> > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager >> not >> >> > > > reachable. >> >> > > > > > Please make sure that the actor is running and its port is >> >> > reachable. >> >> > > > > > at >> >> > > > > > >> >> > > > >> >> > >> >> >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) >> >> > > > > > at >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) >> >> > > > > > ... 14 more >> >> > > > > > Caused by: java.util.concurrent.TimeoutException: Futures >> timed >> >> out >> >> > > > after >> >> > > > > > [10000 milliseconds] >> >> > > > > > at >> >> > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> >> > > > > > at >> >> > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> >> > > > > > at >> >> > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> >> > > > > > at >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> >> > > > > > at scala.concurrent.Await$.result(package.scala:107) >> >> > > > > > at >> >> > > > > > >> >> > > > >> >> > >> >> >> org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) >> >> > > > > > ... 19 more >> >> > > > > > >> >> > > > > > I try ping my jobmanager with >> >> > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the >> >> following >> >> > > as >> >> > > > > > responces. >> >> > > > > > >> >> > > > > > curl: (52) Empty reply from server >> >> > > > > > >> >> > > > > > Which is an indication that the job manager is reachable. >> >> > > > > > >> >> > > > > > So I was wondering if I doing it the right way. Please any >> help >> >> > will >> >> > > be >> >> > > > > > welcoming. >> >> > > > > > >> >> > > > > > Thanks, >> >> > > > > > Ned >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >> > > |
I think then you have to either reconfigure your cluster environment or
wait until we bump the Akka version to 2.4.x which supports having an internal and external IP address. Cheers, Till On Fri, Apr 15, 2016 at 6:36 PM, star jlong <[hidden email]> wrote: > Hi Till/Ned, > > Soory I thought this was my post. > > Le Vendredi 15 avril 2016 17h28, ned dogg <[hidden email]> a > écrit : > > > Hi Till/Jstar, > > Thanks for the the reply. > Well I'm facing the same issue as Jstar. Here is my scenarios, I have app > that is creating flink cluster on VMs for users. This app cannot create > generate a file with the key-pair. So ssh the instance to deploy the flink > topology wouldn't be possible as we can't provide them with key-pairs. > On Fri, Apr 15, 2016 at 4:36 PM, star jlong <[hidden email]> > wrote: > > Hi Till, > > Thanks for the reply. > The idea of ssh of the instance is a good one. I thought of that but in my > case it is not applicable because I setting up a cluster some employees of > a company. And ssh the instance by employees will mean giving them > the instance's key-pair, which I can not. Matter of fact, I'm the one who > is supposed to handle the maintainance of the cluster. > > > > > Le Vendredi 15 avril 2016 10h34, Till Rohrmann <[hidden email]> > a écrit : > > > Hi Ned, > > what you also could do is to ssh to your remote cluster and submit the job > using the private IP address which is reachable from within your cluster. I > don't know whether that would be applicable to your use case. > > Cheers, > Till > > On Fri, Apr 15, 2016 at 9:22 AM, Till Rohrmann <[hidden email]> > wrote: > > > The log says: Unable to allocate on port 6123, due to error: Cannot > assign > > requested address > > > > Thus, I would assume that something with your cluster configuration is > not > > entirely correct. Could you check that? > > > > On Thu, Apr 14, 2016 at 11:19 AM, ned dogg <[hidden email]> wrote: > > > >> Here a try that I given. As first I was configuring my cluster with > >> private > >> ip and it was starting properly. > >> So to avoid this akka issue, I decided to configure my cluster with > public > >> address but with this configuration, my cluster is not starting at all. > >> Here is the logs that I get > >> 2016-04-14 09:08:52,845 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> > >> > -------------------------------------------------------------------------------- > >> 2016-04-14 09:08:52,847 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Starting > >> JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 UTC) > >> 2016-04-14 09:08:52,847 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Current > >> user: ubuntu > >> 2016-04-14 09:08:52,847 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - JVM: > >> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 > >> 2016-04-14 09:08:52,847 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Maximum > >> heap size: 247 MiBytes > >> 2016-04-14 09:08:52,847 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> JAVA_HOME: (not set) > >> 2016-04-14 09:08:52,880 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - Hadoop > >> version: 1.2.1 > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - JVM > >> Options: > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> -Xms256m > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> -Xmx256m > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> -XX:MaxPermSize=256m > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> > >> > -Dlog.file=/home/ubuntu/flink/log/flink-ubuntu-jobmanager-1-tresor-thiswithpublicip.479319055.streamly.com.log > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Program > >> Arguments: > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> --configDir > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> /home/ubuntu/flink/conf > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> --executionMode > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> cluster > >> 2016-04-14 09:08:52,881 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> Classpath: > >> > >> > /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: > >> 2016-04-14 09:08:52,882 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> > >> > -------------------------------------------------------------------------------- > >> 2016-04-14 09:08:52,883 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > >> Registered > >> UNIX signal handlers for [TERM, HUP, INT] > >> 2016-04-14 09:08:53,324 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - Loading > >> configuration from /home/ubuntu/flink/conf > >> 2016-04-14 09:08:53,373 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Starting > >> JobManager without high-availability > >> 2016-04-14 09:08:53,390 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Starting > >> JobManager on 54.233.145.230:6123 with execution mode CLUSTER > >> 2016-04-14 09:08:53,613 INFO > >> org.apache.flink.runtime.jobmanager.JobManager - > Security > >> is not enabled. Starting non-authenticated JobManager. > >> 2016-04-14 09:08:53,651 INFO org.apache.flink.util.NetUtils > >> - Unable to allocate on port 6123, due to error: Cannot > >> assign requested address > >> 2016-04-14 09:08:53,654 ERROR > >> org.apache.flink.runtime.jobmanager.JobManager - Failed > to > >> run JobManager. > >> java.lang.RuntimeException: Unable to do further retries starting the > >> actor > >> system > >> at > >> > >> > org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:1779) > >> at > >> > >> > org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1719) > >> at > >> > >> > org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1655) > >> at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala) > >> > >> > >> > >> On Thu, Apr 14, 2016 at 10:11 AM, ned dogg <[hidden email]> wrote: > >> > >> > Thanks Till for the reply. > >> > > >> > But according to you how can I address that? > >> > > >> > Thanks, > >> > Ned > >> > > >> > On Thu, Apr 14, 2016 at 9:56 AM, Till Rohrmann < > [hidden email] > >> > > >> > wrote: > >> > > >> >> Hi Ned, > >> >> > >> >> I think you are facing the issue described in this JIRA issue [1]. > The > >> >> problem is that you have a private and a public IP address and that > >> Akka > >> >> binds to the private IP address. Since the registered IP of an > >> ActorSystem > >> >> and the target IP address of a request to this ActorSystem have to be > >> >> matching, you cannot reach the ActorSystem via the public IP address. > >> >> Requests with a non-matching IP address are discarded, as indicated > by > >> the > >> >> last log statements. > >> >> > >> >> [1] https://issues.apache.org/jira/browse/FLINK-2821 > >> >> > >> >> Cheers, > >> >> Till > >> >> > >> >> On Thu, Apr 14, 2016 at 10:41 AM, ned dogg <[hidden email]> > >> wrote: > >> >> > >> >> > 2016-04-14 08:23:51,900 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > > >> >> > > >> >> > >> > -------------------------------------------------------------------------------- > >> >> > 2016-04-14 08:23:51,902 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager (Version: 1.0.1, Rev:4afa401, Date:31.03.2016 @ 13:40:33 > >> UTC) > >> >> > 2016-04-14 08:23:51,902 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Current > >> >> > user: root > >> >> > 2016-04-14 08:23:51,902 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> JVM: > >> >> > OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01 > >> >> > 2016-04-14 08:23:51,902 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Maximum > >> >> > heap size: 247 MiBytes > >> >> > 2016-04-14 08:23:51,902 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > JAVA_HOME: /usr/lib/jvm/java-7-openjdk-amd64 > >> >> > 2016-04-14 08:23:51,929 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Hadoop > >> >> > version: 1.2.1 > >> >> > 2016-04-14 08:23:51,929 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > JVM > >> >> > Options: > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > -Xms256m > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > -Xmx256m > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > -XX:MaxPermSize=256m > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > > >> >> > > >> >> > >> > -Dlog.file=/home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > -Dlog4j.configuration=file:/home/ubuntu/flink/conf/log4j.properties > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > > -Dlogback.configurationFile=file:/home/ubuntu/flink/conf/logback.xml > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Program > >> >> > Arguments: > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > --configDir > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > /home/ubuntu/flink/conf > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > --executionMode > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > cluster > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > Classpath: > >> >> > > >> >> > > >> >> > >> > /home/ubuntu/flink/lib/flink-dist_2.10-1.0.1.jar:/home/ubuntu/flink/lib/flink-python_2.10-1.0.1.jar:/home/ubuntu/flink/lib/log4j-1.2.17.jar:/home/ubuntu/flink/lib/slf4j-log4j12-1.7.7.jar::: > >> >> > 2016-04-14 08:23:51,930 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> > > >> >> > > >> >> > >> > -------------------------------------------------------------------------------- > >> >> > 2016-04-14 08:23:51,931 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Registered > >> >> > UNIX signal handlers for [TERM, HUP, INT] > >> >> > 2016-04-14 08:23:52,362 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> Loading > >> >> > configuration from /home/ubuntu/flink/conf > >> >> > 2016-04-14 08:23:52,400 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager without high-availability > >> >> > 2016-04-14 08:23:52,408 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager on 172.31.45.232:6123 with execution mode CLUSTER > >> >> > 2016-04-14 08:23:52,655 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Security > >> >> > is not enabled. Starting non-authenticated JobManager. > >> >> > 2016-04-14 08:23:52,701 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager > >> >> > 2016-04-14 08:23:52,701 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager actor system at 172.31.45.232:6123 > >> >> > 2016-04-14 08:23:54,091 INFO akka.event.slf4j.Slf4jLogger > >> >> > - Slf4jLogger started > >> >> > 2016-04-14 08:23:54,293 INFO Remoting > >> >> > - Starting remoting > >> >> > 2016-04-14 08:23:54,712 INFO Remoting > >> >> > - Remoting started; listening on addresses > >> >> :[akka.tcp:// > >> >> > flink@172.31.45.232:6123] > >> >> > 2016-04-14 08:23:54,732 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManger web frontend > >> >> > 2016-04-14 08:23:54,777 INFO > >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - > >> >> Determined > >> >> > location of JobManager log file: > >> >> > > >> >> > > >> >> > >> > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.log > >> >> > 2016-04-14 08:23:54,777 INFO > >> >> > org.apache.flink.runtime.webmonitor.WebMonitorUtils - > >> >> Determined > >> >> > location of JobManager stdout file: > >> >> > > >> >> > > >> >> > >> > /home/ubuntu/flink/log/flink-root-jobmanager-0-tresor-testflinkth.1541841583.streamly.com.out > >> >> > 2016-04-14 08:23:54,805 INFO > >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - > >> Using > >> >> > directory /tmp/flink-web-dadb12c4-4ac9-42d6-b127-712db44b4add for > the > >> >> web > >> >> > interface files > >> >> > 2016-04-14 08:23:54,805 INFO > >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - > >> Using > >> >> > directory > /tmp/flink-web-upload-88dc8823-f114-4787-a6f4-f1955380e384 > >> for > >> >> > web frontend JAR file uploads > >> >> > 2016-04-14 08:23:55,600 INFO > >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web > >> >> > frontend listening at 0:0:0:0:0:0:0:0:8081 > >> >> > 2016-04-14 08:23:55,601 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager actor > >> >> > 2016-04-14 08:23:55,619 INFO > >> org.apache.flink.runtime.blob.BlobServer > >> >> > - Created BLOB server storage directory > >> >> > /tmp/blobStore-bbc7f33e-fa65-41f8-8c68-549e9707fd56 > >> >> > 2016-04-14 08:23:55,634 INFO > >> org.apache.flink.runtime.blob.BlobServer > >> >> > - Started BLOB server at 0.0.0.0:60439 - max > >> >> concurrent > >> >> > requests: 50 - max backlog: 1000 > >> >> > 2016-04-14 08:23:55,653 INFO > >> >> > org.apache.flink.runtime.checkpoint.SavepointStoreFactory - > >> Using > >> >> job > >> >> > manager savepoint state backend. > >> >> > 2016-04-14 08:23:55,678 INFO > >> >> > org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - > >> >> Starting > >> >> > with JobManager akka.tcp:// > flink@172.31.45.232:6123/user/jobmanager > >> on > >> >> > port > >> >> > 8081 > >> >> > 2016-04-14 08:23:55,678 INFO > >> >> > org.apache.flink.runtime.webmonitor.JobManagerRetriever - New > >> >> leader > >> >> > reachable under akka.tcp:// > >> >> flink@172.31.45.232:6123/user/jobmanager:null. > >> >> > 2016-04-14 08:23:55,692 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> Starting > >> >> > JobManager at akka.tcp://flink@172.31.45.232:6123/user/jobmanager. > >> >> > 2016-04-14 08:23:55,696 INFO > >> >> > org.apache.flink.runtime.jobmanager.MemoryArchivist - > >> Started > >> >> > memory archivist akka://flink/user/archive > >> >> > 2016-04-14 08:23:55,702 INFO > >> >> > org.apache.flink.runtime.jobmanager.JobManager - > >> >> JobManager > >> >> > akka.tcp://flink@172.31.45.232:6123/user/jobmanager was granted > >> >> leadership > >> >> > with leader session ID None. > >> >> > 2016-04-14 08:24:12,740 INFO > >> >> > org.apache.flink.runtime.instance.InstanceManager - > >> >> Registered > >> >> > TaskManager at tresor-testflinkth (akka.tcp:// > >> >> > flink@172.31.45.130:42189/user/taskmanager) as > >> >> > 734b6b21dd60760c7a05722db3c3d1c4. Current number of registered > hosts > >> is > >> >> 1. > >> >> > Current number of alive task slots is 1. > >> >> > 2016-04-14 08:24:17,855 INFO > >> >> > org.apache.flink.runtime.instance.InstanceManager - > >> >> Registered > >> >> > TaskManager at tresor-testflinkth (akka.tcp:// > >> >> > flink@172.31.34.121:58814/user/taskmanager) as > >> >> > fead50f6831aa3f341d58162bb918d90. Current number of registered > hosts > >> is > >> >> 2. > >> >> > Current number of alive task slots is 2. > >> >> > 2016-04-14 08:29:50,346 ERROR akka.remote.EndpointWriter > >> >> > - dropping message [class > >> >> > akka.actor.ActorSelectionMessage] for non-local recipient > >> >> > [Actor[akka.tcp:// > >> >> > flink@54.233.183.228:6123/]] arriving at [akka.tcp:// > >> >> > flink@54.233.183.228:6123] inbound addresses are [akka.tcp:// > >> >> > flink@172.31.45.232:6123] > >> >> > 2016-04-14 08:29:59,777 WARN > akka.remote.ReliableDeliverySupervisor > >> >> > - Association with remote system [akka.tcp:// > >> >> > flink@127.0.0.1:35953] has failed, address is now gated for [5000] > >> ms. > >> >> > Reason is: [Disassociated]. > >> >> > > >> >> > > >> >> > Where 54.233.183.228 <http://flink@54.233.183.228:6123> is the > >> public > >> >> > address of the VM hosting job manager and 172.31.45.232 > >> >> > <http://flink@172.31.45.232:6123> is it private address. > >> >> > 172.31.45.130 <http://flink@172.31.45.130:42189/user/taskmanager> > >> and > >> >> > 172.31.34.121 <http://flink@172.31.34.121:58814/user/taskmanager> > >> are > >> >> the > >> >> > private address of the task manager > >> >> > > >> >> > On Thu, Apr 14, 2016 at 9:05 AM, Till Rohrmann < > [hidden email] > >> > > >> >> > wrote: > >> >> > > >> >> > > I'm referring to the jobmanager.log file not the client log file. > >> You > >> >> can > >> >> > > find it in the `/log` directory. > >> >> > > > >> >> > > Cheers, > >> >> > > Till > >> >> > > > >> >> > > On Thu, Apr 14, 2016 at 9:56 AM, ned dogg <[hidden email]> > >> >> wrote: > >> >> > > > >> >> > > > Hi Till > >> >> > > > > >> >> > > > Thanks for the prompt reply. > >> >> > > > > >> >> > > > The logs say that Please make sure that the actor is running > and > >> its > >> >> > port > >> >> > > > is reachable. > >> >> > > > And it is actaully reachable because I can ping that address. > >> >> > > > > >> >> > > > Ned. > >> >> > > > > >> >> > > > On Thu, Apr 14, 2016 at 8:43 AM, Till Rohrmann < > >> >> > [hidden email]> > >> >> > > > wrote: > >> >> > > > > >> >> > > > > Hi Ned, > >> >> > > > > > >> >> > > > > what does the logs of the JobManager say? > >> >> > > > > > >> >> > > > > Cheers, > >> >> > > > > Till > >> >> > > > > On Apr 14, 2016 9:19 AM, "ned dogg" <[hidden email]> > >> wrote: > >> >> > > > > > >> >> > > > > > Hi everybody, > >> >> > > > > > > >> >> > > > > > I'm Ned, a young and passionte developer of apache > >> >> technologies. I > >> >> > > have > >> >> > > > > > been playing with apache flink lastly. > >> >> > > > > > > >> >> > > > > > This is what I wanted to do submit a flink topology to a > >> remote > >> >> > flink > >> >> > > > > > cluster. The following are the steps that I did. > >> >> > > > > > > >> >> > > > > > - Install flink as a cluster indicated on the link > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://ci.apache.org/projects/flink/flink-docs-master/setup/cluster_setup.html > >> >> > > > > > on three remotes VMs. > >> >> > > > > > - Run the sample WordCountRemoteByClient > >> >> > > > > > < > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > https://github.com/apache/flink/blob/master/flink-contrib/flink-storm-examples/src/main/java/org/apache/flink/storm/wordcount/WordCountRemoteByClient.java > >> >> > > > > > > > >> >> > > > > > by > >> >> > > > > > changing > >> >> > > > > > conf.put(Config.NIMBUS_HOST, "localhost"); to > >> >> > > > > > conf.put(Config.NIMBUS_HOST, > >> >> > > "publicIpOfJobmanagerInMyRemoteCluster"); > >> >> > > > > > > >> >> > > > > > Unfortunately for me when I run that program, I have a the > >> >> > following > >> >> > > > > > exception. > >> >> > > > > > > >> >> > > > > > org.apache.flink.client.program.ProgramInvocationException: > >> The > >> >> > main > >> >> > > > > method > >> >> > > > > > caused an error. > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) > >> >> > > > > > at > >> >> > > > org.apache.flink.client.program.Client.runBlocking(Client.java:248) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866) > >> >> > > > > > at > >> org.apache.flink.client.CliFrontend.run(CliFrontend.java:333) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189) > >> >> > > > > > at > >> >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239) > >> >> > > > > > Caused by: java.lang.RuntimeException: Could not connect to > >> >> Flink > >> >> > > > > > JobManager with address > >> >> publicIpOfJobmanagerInMyRemoteCluster:6123 > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:305) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.storm.api.FlinkClient.submitTopologyWithOpts(FlinkClient.java:177) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.storm.api.FlinkClient.submitTopology(FlinkClient.java:167) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > stormWorldCount.WordCountRemoteByClient.main(WordCountRemoteByClient.java:72) > >> >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> Method) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >> > > > > > at java.lang.reflect.Method.invoke(Method.java:483) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) > >> >> > > > > > ... 6 more > >> >> > > > > > Caused by: java.io.IOException: Actor at akka.tcp://flink@ > >> >> > > > > > publicIpOfJobmanagerInMyRemoteCluster:6123/user/jobmanager > >> not > >> >> > > > reachable. > >> >> > > > > > Please make sure that the actor is running and its port is > >> >> > reachable. > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > >> >> > > >> >> > >> > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:384) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2380) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerActorRef(JobManager.scala:2400) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerActorRef(JobManager.scala) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.storm.api.FlinkClient.getJobManager(FlinkClient.java:333) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > org.apache.flink.storm.api.FlinkClient.getTopologyJobId(FlinkClient.java:279) > >> >> > > > > > ... 14 more > >> >> > > > > > Caused by: java.util.concurrent.TimeoutException: Futures > >> timed > >> >> out > >> >> > > > after > >> >> > > > > > [10000 milliseconds] > >> >> > > > > > at > >> >> > > > > >> >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > >> >> > > > > > at > >> >> > > > > >> >> > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > >> >> > > > > > at > >> >> > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > >> >> > > > > > at scala.concurrent.Await$.result(package.scala:107) > >> >> > > > > > at > >> >> > > > > > > >> >> > > > > >> >> > > >> >> > >> > org.apache.flink.runtime.akka.AkkaUtils$.getActorRef(AkkaUtils.scala:380) > >> >> > > > > > ... 19 more > >> >> > > > > > > >> >> > > > > > I try ping my jobmanager with > >> >> > > > > > curl publicIpOfJobmanagerInMyRemoteCluster:6123 I had the > >> >> following > >> >> > > as > >> >> > > > > > responces. > >> >> > > > > > > >> >> > > > > > curl: (52) Empty reply from server > >> >> > > > > > > >> >> > > > > > Which is an indication that the job manager is reachable. > >> >> > > > > > > >> >> > > > > > So I was wondering if I doing it the right way. Please any > >> help > >> >> > will > >> >> > > be > >> >> > > > > > welcoming. > >> >> > > > > > > >> >> > > > > > Thanks, > >> >> > > > > > Ned > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > > >> > > >> > > > > > > > > > > > > |
Free forum by Nabble | Edit this page |