Hi,
First of all, I wish everybody a happy new year 2017. I've set user@flink in CC so that users who are interested in helping with the testing get notified. Please respond only to the dev@ list to keep the discussion there! According to the 1.2 release discussion thread, I've created a first release candidate for Flink 1.2. The release candidate will not be the final release, because I'm certain that we'll find at least one blocking issue in the candidate :) Therefore, the RC is meant as a testing only release candidate. Please report every issue we need to fix before the next RC in this thread so that we have a good overview. The release artifacts are located here: http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ The maven staging repository is located here: https://repository.apache.org/content/repositories/orgapacheflink-1111 The release commit (in branch "release-1.2.0-rc0"): http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced Happy testing! |
Do we have a probable time of 1.2 release? This month or Next month?
-----邮件原件----- 发件人: Robert Metzger [mailto:[hidden email]] 发送时间: 2017年1月3日 20:44 收件人: [hidden email] 抄送: [hidden email] 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate) Hi, First of all, I wish everybody a happy new year 2017. I've set user@flink in CC so that users who are interested in helping with the testing get notified. Please respond only to the dev@ list to keep the discussion there! According to the 1.2 release discussion thread, I've created a first release candidate for Flink 1.2. The release candidate will not be the final release, because I'm certain that we'll find at least one blocking issue in the candidate :) Therefore, the RC is meant as a testing only release candidate. Please report every issue we need to fix before the next RC in this thread so that we have a good overview. The release artifacts are located here: http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ The maven staging repository is located here: https://repository.apache.org/content/repositories/orgapacheflink-1111 The release commit (in branch "release-1.2.0-rc0"): http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced Happy testing! |
Hi,
this depends a lot on the number of issues we find during the testing. These are the issues I found so far: https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) https://issues.apache.org/jira/browse/FLINK-5383 (resolved) https://issues.apache.org/jira/browse/FLINK-5382 (resolved) https://issues.apache.org/jira/browse/FLINK-5381 (resolved) https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> wrote: > Do we have a probable time of 1.2 release? This month or Next month? > > -----邮件原件----- > 发件人: Robert Metzger [mailto:[hidden email]] > 发送时间: 2017年1月3日 20:44 > 收件人: [hidden email] > 抄送: [hidden email] > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate) > > Hi, > > First of all, I wish everybody a happy new year 2017. > > I've set user@flink in CC so that users who are interested in helping > with the testing get notified. Please respond only to the dev@ list to > keep the discussion there! > > According to the 1.2 release discussion thread, I've created a first > release candidate for Flink 1.2. > The release candidate will not be the final release, because I'm certain > that we'll find at least one blocking issue in the candidate :) > > Therefore, the RC is meant as a testing only release candidate. > Please report every issue we need to fix before the next RC in this thread > so that we have a good overview. > > The release artifacts are located here: > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > > The maven staging repository is located here: > https://repository.apache.org/content/repositories/orgapacheflink-1111 > > The release commit (in branch "release-1.2.0-rc0"): > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > > > Happy testing! > |
I have another bugfix for 1.2.:
https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) 2017-01-10 15:16 GMT+01:00 Robert Metzger <[hidden email]>: > Hi, > > this depends a lot on the number of issues we find during the testing. > > > These are the issues I found so far: > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) > > > > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> wrote: > > > Do we have a probable time of 1.2 release? This month or Next month? > > > > -----邮件原件----- > > 发件人: Robert Metzger [mailto:[hidden email]] > > 发送时间: 2017年1月3日 20:44 > > 收件人: [hidden email] > > 抄送: [hidden email] > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release > candidate) > > > > Hi, > > > > First of all, I wish everybody a happy new year 2017. > > > > I've set user@flink in CC so that users who are interested in helping > > with the testing get notified. Please respond only to the dev@ list to > > keep the discussion there! > > > > According to the 1.2 release discussion thread, I've created a first > > release candidate for Flink 1.2. > > The release candidate will not be the final release, because I'm certain > > that we'll find at least one blocking issue in the candidate :) > > > > Therefore, the RC is meant as a testing only release candidate. > > Please report every issue we need to fix before the next RC in this > thread > > so that we have a good overview. > > > > The release artifacts are located here: > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > > > > The maven staging repository is located here: > > https://repository.apache.org/content/repositories/orgapacheflink-1111 > > > > The release commit (in branch "release-1.2.0-rc0"): > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > > > > > > Happy testing! > > > |
I also found a bunch of issues
https://issues.apache.org/jira/browse/FLINK-5465 https://issues.apache.org/jira/browse/FLINK-5462 https://issues.apache.org/jira/browse/FLINK-5464 https://issues.apache.org/jira/browse/FLINK-5463 On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske <[hidden email]> wrote: > I have another bugfix for 1.2.: > > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) > > 2017-01-10 15:16 GMT+01:00 Robert Metzger <[hidden email]>: > > > Hi, > > > > this depends a lot on the number of issues we find during the testing. > > > > > > These are the issues I found so far: > > > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) > > > > > > > > > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> > wrote: > > > > > Do we have a probable time of 1.2 release? This month or Next month? > > > > > > -----邮件原件----- > > > 发件人: Robert Metzger [mailto:[hidden email]] > > > 发送时间: 2017年1月3日 20:44 > > > 收件人: [hidden email] > > > 抄送: [hidden email] > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release > > candidate) > > > > > > Hi, > > > > > > First of all, I wish everybody a happy new year 2017. > > > > > > I've set user@flink in CC so that users who are interested in helping > > > with the testing get notified. Please respond only to the dev@ list to > > > keep the discussion there! > > > > > > According to the 1.2 release discussion thread, I've created a first > > > release candidate for Flink 1.2. > > > The release candidate will not be the final release, because I'm > certain > > > that we'll find at least one blocking issue in the candidate :) > > > > > > Therefore, the RC is meant as a testing only release candidate. > > > Please report every issue we need to fix before the next RC in this > > thread > > > so that we have a good overview. > > > > > > The release artifacts are located here: > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > > > > > > The maven staging repository is located here: > > > https://repository.apache.org/content/repositories/orgapacheflink-1111 > > > > > > The release commit (in branch "release-1.2.0-rc0"): > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > > > > > > > > > Happy testing! > > > > > > |
I also found an issue:
https://issues.apache.org/jira/browse/FLINK-5470 I also noticed that Flink's webserver does not support https requests. It might be worthwhile to add it, though. https://issues.apache.org/jira/browse/FLINK-5472 On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger <[hidden email]> wrote: > I also found a bunch of issues > > https://issues.apache.org/jira/browse/FLINK-5465 > https://issues.apache.org/jira/browse/FLINK-5462 > https://issues.apache.org/jira/browse/FLINK-5464 > https://issues.apache.org/jira/browse/FLINK-5463 > > > On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske <[hidden email]> wrote: > > > I have another bugfix for 1.2.: > > > > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) > > > > 2017-01-10 15:16 GMT+01:00 Robert Metzger <[hidden email]>: > > > > > Hi, > > > > > > this depends a lot on the number of issues we find during the testing. > > > > > > > > > These are the issues I found so far: > > > > > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) > > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) > > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) > > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) > > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) > > > > > > > > > > > > > > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> > > wrote: > > > > > > > Do we have a probable time of 1.2 release? This month or Next month? > > > > > > > > -----邮件原件----- > > > > 发件人: Robert Metzger [mailto:[hidden email]] > > > > 发送时间: 2017年1月3日 20:44 > > > > 收件人: [hidden email] > > > > 抄送: [hidden email] > > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release > > > candidate) > > > > > > > > Hi, > > > > > > > > First of all, I wish everybody a happy new year 2017. > > > > > > > > I've set user@flink in CC so that users who are interested in > helping > > > > with the testing get notified. Please respond only to the dev@ list > to > > > > keep the discussion there! > > > > > > > > According to the 1.2 release discussion thread, I've created a first > > > > release candidate for Flink 1.2. > > > > The release candidate will not be the final release, because I'm > > certain > > > > that we'll find at least one blocking issue in the candidate :) > > > > > > > > Therefore, the RC is meant as a testing only release candidate. > > > > Please report every issue we need to fix before the next RC in this > > > thread > > > > so that we have a good overview. > > > > > > > > The release artifacts are located here: > > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > > > > > > > > The maven staging repository is located here: > > > > https://repository.apache.org/content/repositories/ > orgapacheflink-1111 > > > > > > > > The release commit (in branch "release-1.2.0-rc0"): > > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > > > > > > > > > > > > Happy testing! > > > > > > > > > > |
I'm wondering whether we should not depend the webserver encryption on the
global encryption activation and activating it instead per default. On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler <[hidden email]> wrote: > FLINK-5470 is a duplicate of FLINK-5298 for which there is also an open PR. > > FLINK-5472 is imo invalid since the webserver does support https, you just > have to enable it as per the security documentation. > > > On 12.01.2017 16:20, Till Rohrmann wrote: > > I also found an issue: > > https://issues.apache.org/jira/browse/FLINK-5470 > > I also noticed that Flink's webserver does not support https requests. It > might be worthwhile to add it, though. > > https://issues.apache.org/jira/browse/FLINK-5472 > > On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger <[hidden email]> > wrote: > >> I also found a bunch of issues >> >> https://issues.apache.org/jira/browse/FLINK-5465 >> https://issues.apache.org/jira/browse/FLINK-5462 >> https://issues.apache.org/jira/browse/FLINK-5464 >> https://issues.apache.org/jira/browse/FLINK-5463 >> >> >> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < <[hidden email]> >> [hidden email]> wrote: >> >> > I have another bugfix for 1.2.: >> > >> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) >> > >> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < <[hidden email]> >> [hidden email]>: >> > >> > > Hi, >> > > >> > > this depends a lot on the number of issues we find during the testing. >> > > >> > > >> > > These are the issues I found so far: >> > > >> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) >> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) >> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) >> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) >> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) >> > > >> > > >> > > >> > > >> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> >> > wrote: >> > > >> > > > Do we have a probable time of 1.2 release? This month or Next month? >> > > > >> > > > -----邮件原件----- >> > > > 发件人: Robert Metzger [mailto: <[hidden email]> >> [hidden email]] >> > > > 发送时间: 2017年1月3日 20:44 >> > > > 收件人: [hidden email] >> > > > 抄送: [hidden email] >> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release >> > > candidate) >> > > > >> > > > Hi, >> > > > >> > > > First of all, I wish everybody a happy new year 2017. >> > > > >> > > > I've set user@flink in CC so that users who are interested in >> helping >> > > > with the testing get notified. Please respond only to the dev@ >> list to >> > > > keep the discussion there! >> > > > >> > > > According to the 1.2 release discussion thread, I've created a first >> > > > release candidate for Flink 1.2. >> > > > The release candidate will not be the final release, because I'm >> > certain >> > > > that we'll find at least one blocking issue in the candidate :) >> > > > >> > > > Therefore, the RC is meant as a testing only release candidate. >> > > > Please report every issue we need to fix before the next RC in this >> > > thread >> > > > so that we have a good overview. >> > > > >> > > > The release artifacts are located here: >> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ >> > > > >> > > > The maven staging repository is located here: >> > > > https://repository.apache.org/content/repositories/orgapache >> flink-1111 >> > > > >> > > > The release commit (in branch "release-1.2.0-rc0"): >> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced >> > > > >> > > > >> > > > Happy testing! >> > > > >> > > >> > >> > > > |
Dear All,
I get an error in jobmanage.log following when I submit a flink job (batch/WordCount.jar) by using command : "./bin/flink run -m 9.96.101.177:39180 ./examples/batch/WordCount.jar". And the flink is on yarn cluster. Error in jobmanage.log : 2017-01-13 15:28:27,402 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@9.96.101.177:39180/]] arriving at [akka.tcp://flink@9.96.101.177:39180] inbound addresses are [akka.tcp://flink@9-96-101-177:39180] However, It is success when I use flink web-ui to submit the job. How to solve this problem? And otherwise, when I started the flink on yarn, the jobmanage.rpc.port and the web port both were changed to 39180 and 57724. The configuration following in flink-conf.yaml is just as default : jobmanager.rpc.port: 6123 and jobmanager.web.port: 8081 I started the flink on yarn using command : "./bin/yarn-session.sh -n 4". Why were the ports changed to 39180 and 57724? Many thanks if there is any help! HuangWHWHW 2017.1.13 |
Hi Huang,
this seems to be very strange, because the JobManager’s actor system has bound to the address 9-96-101-177 instead of 9.96.101.177. It seems a if the . have been replaced by -. Could you maybe tell me which version of Flink you’re running and also share the complete JobManager log with us? I tested it with the latest 1.2 SNAPSHOT version and there it seemed to work. Cheers, Till On Fri, Jan 13, 2017 at 9:02 AM, huangwei (G) <[hidden email]> wrote: > Dear All, > > I get an error in jobmanage.log following when I submit a flink job > (batch/WordCount.jar) by using command : "./bin/flink run -m > 9.96.101.177:39180 ./examples/batch/WordCount.jar". > > And the flink is on yarn cluster. > > Error in jobmanage.log : > 2017-01-13 15:28:27,402 ERROR akka.remote.EndpointWriter > - dropping message [class akka.actor.ActorSelectionMessage] > for non-local recipient [Actor[akka.tcp://flink@9.96.101.177:39180/]] > arriving at [akka.tcp://flink@9.96.101.177:39180] inbound addresses are > [akka.tcp://flink@9-96-101-177:39180] > > However, It is success when I use flink web-ui to submit the job. > > How to solve this problem? > > And otherwise, when I started the flink on yarn, the jobmanage.rpc.port > and the web port both were changed to 39180 and 57724. > The configuration following in flink-conf.yaml is just as default : > > jobmanager.rpc.port: 6123 > > and > > jobmanager.web.port: 8081 > > I started the flink on yarn using command : "./bin/yarn-session.sh -n 4". > > Why were the ports changed to 39180 and 57724? > > Many thanks if there is any help! > > HuangWHWHW > 2017.1.13 > |
In reply to this post by Till Rohrmann
I tested the Table API / SQL a bit.
I implemented a windowed aggregation with the streaming Table API and it produced the same results as a DataStream API implementation. Joining a stream with a TableFunction also seemed to work well. Moreover, I checked the results of a bunch of TPC-H queries (batch SQL) and all produced correct results. 2017-01-12 17:45 GMT+01:00 Till Rohrmann <[hidden email]>: > I'm wondering whether we should not depend the webserver encryption on the > global encryption activation and activating it instead per default. > > On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler <[hidden email]> > wrote: > > > FLINK-5470 is a duplicate of FLINK-5298 for which there is also an open > PR. > > > > FLINK-5472 is imo invalid since the webserver does support https, you > just > > have to enable it as per the security documentation. > > > > > > On 12.01.2017 16:20, Till Rohrmann wrote: > > > > I also found an issue: > > > > https://issues.apache.org/jira/browse/FLINK-5470 > > > > I also noticed that Flink's webserver does not support https requests. It > > might be worthwhile to add it, though. > > > > https://issues.apache.org/jira/browse/FLINK-5472 > > > > On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger <[hidden email]> > > wrote: > > > >> I also found a bunch of issues > >> > >> https://issues.apache.org/jira/browse/FLINK-5465 > >> https://issues.apache.org/jira/browse/FLINK-5462 > >> https://issues.apache.org/jira/browse/FLINK-5464 > >> https://issues.apache.org/jira/browse/FLINK-5463 > >> > >> > >> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < <[hidden email]> > >> [hidden email]> wrote: > >> > >> > I have another bugfix for 1.2.: > >> > > >> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) > >> > > >> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < <[hidden email]> > >> [hidden email]>: > >> > > >> > > Hi, > >> > > > >> > > this depends a lot on the number of issues we find during the > testing. > >> > > > >> > > > >> > > These are the issues I found so far: > >> > > > >> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) > >> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) > >> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) > >> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) > >> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) > >> > > > >> > > > >> > > > >> > > > >> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> > >> > wrote: > >> > > > >> > > > Do we have a probable time of 1.2 release? This month or Next > month? > >> > > > > >> > > > -----邮件原件----- > >> > > > 发件人: Robert Metzger [mailto: <[hidden email]> > >> [hidden email]] > >> > > > 发送时间: 2017年1月3日 20:44 > >> > > > 收件人: [hidden email] > >> > > > 抄送: [hidden email] > >> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release > >> > > candidate) > >> > > > > >> > > > Hi, > >> > > > > >> > > > First of all, I wish everybody a happy new year 2017. > >> > > > > >> > > > I've set user@flink in CC so that users who are interested in > >> helping > >> > > > with the testing get notified. Please respond only to the dev@ > >> list to > >> > > > keep the discussion there! > >> > > > > >> > > > According to the 1.2 release discussion thread, I've created a > first > >> > > > release candidate for Flink 1.2. > >> > > > The release candidate will not be the final release, because I'm > >> > certain > >> > > > that we'll find at least one blocking issue in the candidate :) > >> > > > > >> > > > Therefore, the RC is meant as a testing only release candidate. > >> > > > Please report every issue we need to fix before the next RC in > this > >> > > thread > >> > > > so that we have a good overview. > >> > > > > >> > > > The release artifacts are located here: > >> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > >> > > > > >> > > > The maven staging repository is located here: > >> > > > https://repository.apache.org/content/repositories/orgapache > >> flink-1111 > >> > > > > >> > > > The release commit (in branch "release-1.2.0-rc0"): > >> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > >> > > > > >> > > > > >> > > > Happy testing! > >> > > > > >> > > > >> > > >> > > > > > > > |
In reply to this post by Till Rohrmann
Hi Till,
The "9-96-101-177" is just the hostname. I rerun the flink on yarn and here is the jobmanager.log, and sorry for I blocked some sensitive log. By the way , the port(another questions in my earlier mail) seems to be a random value(this time is 38785 and 35699). I used flink-1.2.0, it works well on the yarn which is provided by apache open source. But I run flink on a special yarn which was token some safety reinforcement based on apache-yarn. I just have no idea about the ERROR log. Jobmanage.log: 2017-01-14 09:24:35,584 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -------------------------------------------------------------------------------- 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting YARN ApplicationMaster / ResourceManager / JobManager (Version: 1.2.0, Rev:82b1079, Date:04.01.2017 @ 17:38:23 CST) 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Current user: admin 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.112-b15 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Maximum heap size: 406 MiBytes 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JAVA_HOME: /opt/huawei/Bigdata/jdk1.8.0_112/ 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Hadoop version: 2.7.2 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - JVM Options: 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Xmx424M 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlog.file=/srv/BigData/hadoop/data1/nm/containerlogs/application_1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.log 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlogback.configurationFile=file:logback.xml 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -Dlog4j.configuration=file:log4j.properties 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Program Arguments: (none) 2017-01-14 09:24:35,589 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - -------------------------------------------------------------------------------- 2017-01-14 09:24:35,589 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Registered UNIX signal handlers for [TERM, HUP, INT] 2017-01-14 09:24:35,591 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - remoteKeytabPrincipal obtained admin 2017-01-14 09:24:35,592 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN daemon is running as: admin Yarn client user obtainer: [hidden email] 2017-01-14 09:24:35,596 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Loading config from directory /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/appcache/application_1483499303549_0043/container_1483499303549_0043_01_000001 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 9.96.101.32 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 256 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 512 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.keytab, /home/demo/flink/release/flink-1.2.0/keytab/user.keytab 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.principal, admin 2017-01-14 09:24:35,608 INFO org.apache.flink.runtime.security.JaasConfiguration - Initializing JAAS configuration instance. Parameters: /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/appcache/application_1483499303549_0043/container_1483499303549_0043_01_000001/krb5.keytab, admin 2017-01-14 09:24:35,609 INFO org.apache.flink.runtime.security.SecurityUtils - SASL client auth for ZK will be disabled 2017-01-14 09:24:35,824 INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user admin using keytab file /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/appcache/application_1483499303549_0043/container_1483499303549_0043_01_000001/krb5.keytab 2017-01-14 09:24:35,825 INFO org.apache.flink.runtime.security.SecurityUtils - Hadoop user set to [hidden email] (auth:KERBEROS) 2017-01-14 09:24:35,936 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN assigned hostname for application master: 9-96-101-251 2017-01-14 09:24:35,936 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - keytabPath: /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/appcache/application_1483499303549_0043/container_1483499303549_0043_01_000001/krb5.keytab 2017-01-14 09:24:35,938 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TaskManagers will be created with 1 task slots 2017-01-14 09:24:35,938 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TaskManagers will be started with container size 1024 MB, JVM heap size 424 MB, JVM direct memory limit 424 MB 2017-01-14 09:24:35,943 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Trying to start actor system at 9.96.101.251:38785 2017-01-14 09:24:36,336 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2017-01-14 09:24:36,438 INFO Remoting - Starting remoting 2017-01-14 09:24:36,547 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@9-96-101-251:38785] 2017-01-14 09:24:36,551 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Actor system started at 9.96.101.251:38785 2017-01-14 09:24:36,551 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Actor system bound to hostname 9-96-101-251. 2017-01-14 09:24:36,554 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Setting up resources for TaskManagers 2017-01-14 09:24:36,554 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remoteKeytabPath obtained hdfs://hacluster/user/admin/.flink/application_1483499303549_0043/user.keytab 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remoteKeytabPrincipal obtained admin 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remoteYarnConfPath obtained null 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - TM:remotekrb5Path obtained null 2017-01-14 09:24:36,932 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-01-14 09:24:36,945 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2017-01-14 09:24:36,949 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Adding keytab hdfs://hacluster/user/admin/.flink/application_1483499303549_0043/user.keytab to the AM container local resource bucket 2017-01-14 09:24:37,085 INFO org.apache.flink.yarn.Utils - Copying from file:/srv/BigData/hadoop/data1/nm/localdir/usercache/admin/appcache/application_1483499303549_0043/container_1483499303549_0043_01_000001/e2943789-d80a-4abd-8ae9-2fc14cb1fd03-taskmanager-conf.yaml to hdfs://hacluster/user/admin/.flink/application_1483499303549_0043/e2943789-d80a-4abd-8ae9-2fc14cb1fd03-taskmanager-conf.yaml 2017-01-14 09:24:37,258 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "hacluster" port: -1 file: "/user/admin/.flink/application_1483499303549_0043/e2943789-d80a-4abd-8ae9-2fc14cb1fd03-taskmanager-conf.yaml" } size: 878 timestamp: 1484357077250 type: FILE visibility: APPLICATION 2017-01-14 09:24:37,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Creating container launch context for TaskManagers 2017-01-14 09:24:37,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting TaskManagers with command: $JAVA_HOME/bin/java -Xms424m -Xmx424m -XX:MaxDirectMemorySize=424m -Dlog.file=<LOG_DIR>/taskmanager.log -Dlogback.configurationFile=file:./logback.xml -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager --configDir . 1> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err 2017-01-14 09:24:37,288 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-f91bad88-1473-4e86-b151-f93dffa58baa 2017-01-14 09:24:37,289 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:53972 - max concurrent requests: 50 - max backlog: 1000 2017-01-14 09:24:37,302 INFO org.apache.flink.runtime.metrics.MetricRegistry - No metrics reporter configured, no metrics will be exposed/reported. 2017-01-14 09:24:37,307 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Starting JobManager Web Frontend 2017-01-14 09:24:37,310 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/$a 2017-01-14 09:24:37,311 INFO org.apache.flink.yarn.YarnJobManager - Starting JobManager at akka.tcp://flink@9-96-101-251:38785/user/jobmanager. 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /srv/BigData/hadoop/data1/nm/containerlogs/application_1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.log 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /srv/BigData/hadoop/data1/nm/containerlogs/application_1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.out 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-c4991b46-e637-4207-80ed-caef4cf5702e for the web interface files 2017-01-14 09:24:37,364 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-f8b7b6d7-b066-48a1-9536-40a3cdf42778 for web frontend JAR file uploads 2017-01-14 09:24:37,378 INFO org.apache.flink.yarn.YarnJobManager - JobManager akka.tcp://flink@9-96-101-251:38785/user/jobmanager was granted leadership with leader session ID None. 2017-01-14 09:24:37,560 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0:0:0:0:0:0:0:0:35699 2017-01-14 09:24:37,561 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink@9-96-101-251:38785/user/jobmanager on port 35699 2017-01-14 09:24:37,561 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka://flink/user/jobmanager#-640052308:null. 2017-01-14 09:24:37,568 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN application tolerates 4 failed TaskManager containers before giving up 2017-01-14 09:24:37,571 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - YARN Application Master started 2017-01-14 09:24:37,579 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Initializing YARN resource master 2017-01-14 09:24:37,605 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2017-01-14 09:24:37,606 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Registering Application Master with tracking url http://9-96-101-251:35699 2017-01-14 09:24:37,641 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Trying to associate with JobManager leader akka://flink/user/jobmanager#-640052308 2017-01-14 09:24:37,647 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager#-640052308] - leader session null 2017-01-14 09:24:37,648 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 1024 megabytes memory. Pending requests: 1 2017-01-14 09:24:37,654 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 1024 megabytes memory. Pending requests: 2 2017-01-14 09:24:37,654 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 1024 megabytes memory. Pending requests: 3 2017-01-14 09:24:37,655 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Requesting new TaskManager container with 1024 megabytes memory. Pending requests: 4 2017-01-14 09:24:38,681 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : 9-96-101-177:26009 2017-01-14 09:24:38,692 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1483499303549_0043_01_000002 - Remaining pending container requests: 3 2017-01-14 09:24:38,693 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1484357078692: Container: [ContainerId: container_1483499303549_0043_01_000002, NodeId: 9-96-101-177:26009, NodeHttpAddress: 9-96-101-177:26010, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 9.96.101.177:26009 }, ] on host 9-96-101-177 2017-01-14 09:24:38,694 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : 9-96-101-177:26009 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : 9-96-101-251:26009 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : 9-96-101-32:26009 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : 9-91-8-160:26009 2017-01-14 09:24:39,190 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1483499303549_0043_01_000003 - Remaining pending container requests: 2 2017-01-14 09:24:39,190 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1484357079190: Container: [ContainerId: container_1483499303549_0043_01_000003, NodeId: 9-96-101-251:26009, NodeHttpAddress: 9-96-101-251:26010, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 9.96.101.251:26009 }, ] on host 9-96-101-251 2017-01-14 09:24:39,190 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : 9-96-101-251:26009 2017-01-14 09:24:39,202 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1483499303549_0043_01_000004 - Remaining pending container requests: 1 2017-01-14 09:24:39,202 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1484357079202: Container: [ContainerId: container_1483499303549_0043_01_000004, NodeId: 9-96-101-32:26009, NodeHttpAddress: 9-96-101-32:26010, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 9.96.101.32:26009 }, ] on host 9-96-101-32 2017-01-14 09:24:39,202 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : 9-96-101-32:26009 2017-01-14 09:24:39,217 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1483499303549_0043_01_000005 - Remaining pending container requests: 0 2017-01-14 09:24:39,217 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1484357079217: Container: [ContainerId: container_1483499303549_0043_01_000005, NodeId: 9-91-8-160:26009, NodeHttpAddress: 9-91-8-160:26010, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 9.91.8.160:26009 }, ] on host 9-91-8-160 2017-01-14 09:24:39,217 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : 9-91-8-160:26009 2017-01-14 09:24:43,348 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1483499303549_0043_01_000003 has started. 2017-01-14 09:24:43,350 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at 9-96-101-251 (akka.tcp://flink@9-96-101-251:57010/user/taskmanager) as 49800ab8cfcd1a11e45084a48281df75. Current number of registered hosts is 1. Current number of alive task slots is 1. 2017-01-14 09:24:44,881 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1483499303549_0043_01_000002 has started. 2017-01-14 09:24:44,881 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at 9-96-101-177 (akka.tcp://flink@9-96-101-177:35778/user/taskmanager) as 5d65baf1ec196cf3ac5bc43870156855. Current number of registered hosts is 2. Current number of alive task slots is 2. 2017-01-14 09:24:45,855 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1483499303549_0043_01_000004 has started. 2017-01-14 09:24:45,855 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at 9-96-101-32 (akka.tcp://flink@9-96-101-32:58486/user/taskmanager) as 80de4a47fa60536b78ea052cbadec7ee. Current number of registered hosts is 3. Current number of alive task slots is 3. 2017-01-14 09:24:46,018 INFO org.apache.flink.yarn.YarnFlinkResourceManager - TaskManager container_1483499303549_0043_01_000005 has started. 2017-01-14 09:24:46,018 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at 9-91-8-160 (akka.tcp://flink@9-91-8-160:47548/user/taskmanager) as e9f2497d6223b2d704b3aced665a3c02. Current number of registered hosts is 4. Current number of alive task slots is 4. 2017-01-14 09:29:58,066 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@9.96.101.251:38785/]] arriving at [akka.tcp://flink@9.96.101.251:38785] inbound addresses are [akka.tcp://flink@9-96-101-251:38785] Thanks! HuangWHWHW 2017/1/14 -----邮件原件----- 发件人: Till Rohrmann [mailto:[hidden email]] 发送时间: 2017年1月13日 18:22 收件人: [hidden email] 抄送: [hidden email] 主题: Re: some questions about submit flink job on flink-yarn Hi Huang, this seems to be very strange, because the JobManager’s actor system has bound to the address 9-96-101-177 instead of 9.96.101.177. It seems a if the . have been replaced by -. Could you maybe tell me which version of Flink you’re running and also share the complete JobManager log with us? I tested it with the latest 1.2 SNAPSHOT version and there it seemed to work. Cheers, Till On Fri, Jan 13, 2017 at 9:02 AM, huangwei (G) <[hidden email]> wrote: > Dear All, > > I get an error in jobmanage.log following when I submit a flink job > (batch/WordCount.jar) by using command : "./bin/flink run -m > 9.96.101.177:39180 ./examples/batch/WordCount.jar". > > And the flink is on yarn cluster. > > Error in jobmanage.log : > 2017-01-13 15:28:27,402 ERROR akka.remote.EndpointWriter > - dropping message [class > akka.actor.ActorSelectionMessage] for non-local recipient > [Actor[akka.tcp://flink@9.96.101.177:39180/]] > arriving at [akka.tcp://flink@9.96.101.177:39180] inbound addresses > are [akka.tcp://flink@9-96-101-177:39180] > > However, It is success when I use flink web-ui to submit the job. > > How to solve this problem? > > And otherwise, when I started the flink on yarn, the > jobmanage.rpc.port and the web port both were changed to 39180 and 57724. > The configuration following in flink-conf.yaml is just as default : > > jobmanager.rpc.port: 6123 > > and > > jobmanager.web.port: 8081 > > I started the flink on yarn using command : "./bin/yarn-session.sh -n 4". > > Why were the ports changed to 39180 and 57724? > > Many thanks if there is any help! > > HuangWHWHW > 2017.1.13 > |
Hi Huang,
the reason why you cannot use the IP address to send messages to your YARN JobManager is that we no longer resolve the hostname into an IP address. Instead we start the ActorSystem with the unresolved hostname. You can see this in the following log line: `Actor system bound to hostname 9-96-101-251`. Since Akka requires that the destination address of a message matches exactly the address to which the ActorSystem is bound, you have to use `9-96-101-251:38785`. This was recently changed. Concerning the ports, Flink chooses a random port for the `JobManager` in order to avoid port conflicts with other `JobManagers` running on the same node. With YARN you don't have control over where the `JobManager` is placed. However, you can use the configuration parameter `yarn.application-master.port` to specify a port or a port range for the application master/job manager. Additionally, the web frontends port is always overwritten and set to 0 which means random port selection when starting a yarn session. I hope this clarifies things a little bit. Cheers, Till On Sat, Jan 14, 2017 at 2:52 AM, huangwei (G) <[hidden email]> wrote: > Hi Till, > > The "9-96-101-177" is just the hostname. > I rerun the flink on yarn and here is the jobmanager.log, and sorry for I > blocked some sensitive log. By the way , the port(another questions in my > earlier mail) seems to be a random value(this time is 38785 and 35699). > I used flink-1.2.0, it works well on the yarn which is provided by apache > open source. But I run flink on a special yarn which was token some safety > reinforcement based on apache-yarn. I just have no idea about the ERROR log. > > Jobmanage.log: > > 2017-01-14 09:24:35,584 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - ------------------------------------------------------------ > -------------------- > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Starting YARN ApplicationMaster / ResourceManager / > JobManager (Version: 1.2.0, Rev:82b1079, Date:04.01.2017 @ 17:38:23 CST) > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Current user: admin > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - > 1.8/25.112-b15 > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Maximum heap size: 406 MiBytes > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - JAVA_HOME: /opt/huawei/Bigdata/jdk1.8.0_112/ > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Hadoop version: 2.7.2 > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - JVM Options: > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Xmx424M > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Dlog.file=/srv/BigData/hadoop/data1/nm/containerlogs/ > application_1483499303549_0043/container_1483499303549_ > 0043_01_000001/jobmanager.log > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Dlogback.configurationFile=file:logback.xml > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Dlog4j.configuration=file:log4j.properties > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Program Arguments: (none) > 2017-01-14 09:24:35,589 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - ------------------------------------------------------------ > -------------------- > 2017-01-14 09:24:35,589 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Registered UNIX signal handlers for [TERM, HUP, INT] > 2017-01-14 09:24:35,591 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - remoteKeytabPrincipal obtained admin > 2017-01-14 09:24:35,592 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN daemon is running as: admin Yarn client user obtainer: > [hidden email] > 2017-01-14 09:24:35,596 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Loading config from directory /srv/BigData/hadoop/data1/nm/ > localdir/usercache/admin/appcache/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.rpc.address, > 9.96.101.32 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.rpc.port, 6123 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.heap.mb, 256 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: taskmanager.heap.mb, 512 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: taskmanager.numberOfTaskSlots, 1 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: taskmanager.memory.preallocate, > false > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: parallelism.default, 1 > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.web.port, 8081 > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: security.keytab, > /home/demo/flink/release/flink-1.2.0/keytab/user.keytab > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: security.principal, admin > 2017-01-14 09:24:35,608 INFO org.apache.flink.runtime.security.JaasConfiguration > - Initializing JAAS configuration instance. Parameters: > /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/ > appcache/application_1483499303549_0043/container_ > 1483499303549_0043_01_000001/krb5.keytab, admin > 2017-01-14 09:24:35,609 INFO org.apache.flink.runtime.security.SecurityUtils > - SASL client auth for ZK will be disabled > 2017-01-14 09:24:35,824 INFO org.apache.hadoop.security.UserGroupInformation > - Login successful for user admin using keytab file > /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/ > appcache/application_1483499303549_0043/container_ > 1483499303549_0043_01_000001/krb5.keytab > 2017-01-14 09:24:35,825 INFO org.apache.flink.runtime.security.SecurityUtils > - Hadoop user set to [hidden email] (auth:KERBEROS) > 2017-01-14 09:24:35,936 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN assigned hostname for application master: 9-96-101-251 > 2017-01-14 09:24:35,936 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - keytabPath: /srv/BigData/hadoop/data1/nm/ > localdir/usercache/admin/appcache/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/krb5.keytab > 2017-01-14 09:24:35,938 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TaskManagers will be created with 1 task slots > 2017-01-14 09:24:35,938 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TaskManagers will be started with container size 1024 MB, JVM > heap size 424 MB, JVM direct memory limit 424 MB > 2017-01-14 09:24:35,943 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Trying to start actor system at 9.96.101.251:38785 > 2017-01-14 09:24:36,336 INFO akka.event.slf4j.Slf4jLogger > - Slf4jLogger started > 2017-01-14 09:24:36,438 INFO Remoting > - Starting remoting > 2017-01-14 09:24:36,547 INFO Remoting > - Remoting started; listening on addresses > :[akka.tcp://flink@9-96-101-251:38785] > 2017-01-14 09:24:36,551 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Actor system started at 9.96.101.251:38785 > 2017-01-14 09:24:36,551 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Actor system bound to hostname 9-96-101-251. > 2017-01-14 09:24:36,554 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Setting up resources for TaskManagers > 2017-01-14 09:24:36,554 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remoteKeytabPath obtained hdfs://hacluster/user/admin/. > flink/application_1483499303549_0043/user.keytab > 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remoteKeytabPrincipal obtained admin > 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remoteYarnConfPath obtained null > 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remotekrb5Path obtained null > 2017-01-14 09:24:36,932 WARN org.apache.hadoop.util.NativeCodeLoader > - Unable to load native-hadoop library for your > platform... using builtin-java classes where applicable > 2017-01-14 09:24:36,945 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory > - The short-circuit local reads feature cannot be used because > libhadoop cannot be loaded. > 2017-01-14 09:24:36,949 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Adding keytab hdfs://hacluster/user/admin/.flink/application_1483499303549_0043/user.keytab > to the AM container local resource bucket > 2017-01-14 09:24:37,085 INFO org.apache.flink.yarn.Utils > - Copying from file:/srv/BigData/hadoop/ > data1/nm/localdir/usercache/admin/appcache/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/ > e2943789-d80a-4abd-8ae9-2fc14cb1fd03-taskmanager-conf.yaml to > hdfs://hacluster/user/admin/.flink/application_ > 1483499303549_0043/e2943789-d80a-4abd-8ae9-2fc14cb1fd03- > taskmanager-conf.yaml > 2017-01-14 09:24:37,258 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Prepared local resource for modified yaml: resource { scheme: > "hdfs" host: "hacluster" port: -1 file: "/user/admin/.flink/ > application_1483499303549_0043/e2943789-d80a-4abd-8ae9- > 2fc14cb1fd03-taskmanager-conf.yaml" } size: 878 timestamp: 1484357077250 > type: FILE visibility: APPLICATION > 2017-01-14 09:24:37,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Creating container launch context for TaskManagers > 2017-01-14 09:24:37,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Starting TaskManagers with command: $JAVA_HOME/bin/java > -Xms424m -Xmx424m -XX:MaxDirectMemorySize=424m -Dlog.file=<LOG_DIR>/taskmanager.log > -Dlogback.configurationFile=file:./logback.xml > -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager > --configDir . 1> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err > 2017-01-14 09:24:37,288 INFO org.apache.flink.runtime.blob.BlobServer > - Created BLOB server storage directory > /tmp/blobStore-f91bad88-1473-4e86-b151-f93dffa58baa > 2017-01-14 09:24:37,289 INFO org.apache.flink.runtime.blob.BlobServer > - Started BLOB server at 0.0.0.0:53972 - max concurrent > requests: 50 - max backlog: 1000 > 2017-01-14 09:24:37,302 INFO org.apache.flink.runtime.metrics.MetricRegistry > - No metrics reporter configured, no metrics will be > exposed/reported. > 2017-01-14 09:24:37,307 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Starting JobManager Web Frontend > 2017-01-14 09:24:37,310 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist > - Started memory archivist akka://flink/user/$a > 2017-01-14 09:24:37,311 INFO org.apache.flink.yarn.YarnJobManager > - Starting JobManager at akka.tcp://flink@9-96-101-251: > 38785/user/jobmanager. > 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils > - Determined location of JobManager log file: > /srv/BigData/hadoop/data1/nm/containerlogs/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.log > 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils > - Determined location of JobManager stdout file: > /srv/BigData/hadoop/data1/nm/containerlogs/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.out > 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Using directory /tmp/flink-web-c4991b46-e637-4207-80ed-caef4cf5702e > for the web interface files > 2017-01-14 09:24:37,364 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Using directory /tmp/flink-web-f8b7b6d7-b066-48a1-9536-40a3cdf42778 > for web frontend JAR file uploads > 2017-01-14 09:24:37,378 INFO org.apache.flink.yarn.YarnJobManager > - JobManager akka.tcp://flink@9-96-101-251:38785/user/jobmanager > was granted leadership with leader session ID None. > 2017-01-14 09:24:37,560 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Web frontend listening at 0:0:0:0:0:0:0:0:35699 > 2017-01-14 09:24:37,561 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Starting with JobManager akka.tcp://flink@9-96-101-251:38785/user/jobmanager > on port 35699 > 2017-01-14 09:24:37,561 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever > - New leader reachable under akka://flink/user/jobmanager#- > 640052308:null. > 2017-01-14 09:24:37,568 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN application tolerates 4 failed TaskManager containers > before giving up > 2017-01-14 09:24:37,571 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN Application Master started > 2017-01-14 09:24:37,579 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Initializing YARN resource master > 2017-01-14 09:24:37,605 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies > : 0 > 2017-01-14 09:24:37,606 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Registering Application Master with tracking url > http://9-96-101-251:35699 > 2017-01-14 09:24:37,641 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Trying to associate with JobManager leader > akka://flink/user/jobmanager#-640052308 > 2017-01-14 09:24:37,647 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Resource Manager associating with leading JobManager > Actor[akka://flink/user/jobmanager#-640052308] - leader session null > 2017-01-14 09:24:37,648 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 megabytes > memory. Pending requests: 1 > 2017-01-14 09:24:37,654 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 megabytes > memory. Pending requests: 2 > 2017-01-14 09:24:37,654 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 megabytes > memory. Pending requests: 3 > 2017-01-14 09:24:37,655 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 megabytes > memory. Pending requests: 4 > 2017-01-14 09:24:38,681 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-96-101-177:26009 > 2017-01-14 09:24:38,692 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: container_1483499303549_0043_01_000002 > - Remaining pending container requests: 3 > 2017-01-14 09:24:38,693 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357078692: Container: [ContainerId: container_1483499303549_0043_01_000002, > NodeId: 9-96-101-177:26009, NodeHttpAddress: 9-96-101-177:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, > service: 9.96.101.177:26009 }, ] on host 9-96-101-177 > 2017-01-14 09:24:38,694 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-96-101-177:26009 > 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-96-101-251:26009 > 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-96-101-32:26009 > 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-91-8-160:26009 > 2017-01-14 09:24:39,190 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: container_1483499303549_0043_01_000003 > - Remaining pending container requests: 2 > 2017-01-14 09:24:39,190 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357079190: Container: [ContainerId: container_1483499303549_0043_01_000003, > NodeId: 9-96-101-251:26009, NodeHttpAddress: 9-96-101-251:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, > service: 9.96.101.251:26009 }, ] on host 9-96-101-251 > 2017-01-14 09:24:39,190 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-96-101-251:26009 > 2017-01-14 09:24:39,202 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: container_1483499303549_0043_01_000004 > - Remaining pending container requests: 1 > 2017-01-14 09:24:39,202 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357079202: Container: [ContainerId: container_1483499303549_0043_01_000004, > NodeId: 9-96-101-32:26009, NodeHttpAddress: 9-96-101-32:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, > service: 9.96.101.32:26009 }, ] on host 9-96-101-32 > 2017-01-14 09:24:39,202 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-96-101-32:26009 > 2017-01-14 09:24:39,217 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: container_1483499303549_0043_01_000005 > - Remaining pending container requests: 0 > 2017-01-14 09:24:39,217 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357079217: Container: [ContainerId: container_1483499303549_0043_01_000005, > NodeId: 9-91-8-160:26009, NodeHttpAddress: 9-91-8-160:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, > service: 9.91.8.160:26009 }, ] on host 9-91-8-160 > 2017-01-14 09:24:39,217 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-91-8-160:26009 > 2017-01-14 09:24:43,348 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000003 has > started. > 2017-01-14 09:24:43,350 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-96-101-251 > (akka.tcp://flink@9-96-101-251:57010/user/taskmanager) as > 49800ab8cfcd1a11e45084a48281df75. Current number of registered hosts is > 1. Current number of alive task slots is 1. > 2017-01-14 09:24:44,881 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000002 has > started. > 2017-01-14 09:24:44,881 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-96-101-177 > (akka.tcp://flink@9-96-101-177:35778/user/taskmanager) as > 5d65baf1ec196cf3ac5bc43870156855. Current number of registered hosts is > 2. Current number of alive task slots is 2. > 2017-01-14 09:24:45,855 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000004 has > started. > 2017-01-14 09:24:45,855 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-96-101-32 > (akka.tcp://flink@9-96-101-32:58486/user/taskmanager) as > 80de4a47fa60536b78ea052cbadec7ee. Current number of registered hosts is > 3. Current number of alive task slots is 3. > 2017-01-14 09:24:46,018 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000005 has > started. > 2017-01-14 09:24:46,018 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-91-8-160 > (akka.tcp://flink@9-91-8-160:47548/user/taskmanager) as > e9f2497d6223b2d704b3aced665a3c02. Current number of registered hosts is > 4. Current number of alive task slots is 4. > 2017-01-14 09:29:58,066 ERROR akka.remote.EndpointWriter > - dropping message [class akka.actor.ActorSelectionMessage] > for non-local recipient [Actor[akka.tcp://flink@9.96.101.251:38785/]] > arriving at [akka.tcp://flink@9.96.101.251:38785] inbound addresses are > [akka.tcp://flink@9-96-101-251:38785] > > Thanks! > > HuangWHWHW > 2017/1/14 > > -----邮件原件----- > 发件人: Till Rohrmann [mailto:[hidden email]] > 发送时间: 2017年1月13日 18:22 > 收件人: [hidden email] > 抄送: [hidden email] > 主题: Re: some questions about submit flink job on flink-yarn > > Hi Huang, > > this seems to be very strange, because the JobManager’s actor system has > bound to the address 9-96-101-177 instead of 9.96.101.177. It seems a if > the . have been replaced by -. > > Could you maybe tell me which version of Flink you’re running and also > share the complete JobManager log with us? > > I tested it with the latest 1.2 SNAPSHOT version and there it seemed to > work. > > Cheers, > Till > > > On Fri, Jan 13, 2017 at 9:02 AM, huangwei (G) <[hidden email]> > wrote: > > > Dear All, > > > > I get an error in jobmanage.log following when I submit a flink job > > (batch/WordCount.jar) by using command : "./bin/flink run -m > > 9.96.101.177:39180 ./examples/batch/WordCount.jar". > > > > And the flink is on yarn cluster. > > > > Error in jobmanage.log : > > 2017-01-13 15:28:27,402 ERROR akka.remote.EndpointWriter > > - dropping message [class > > akka.actor.ActorSelectionMessage] for non-local recipient > > [Actor[akka.tcp://flink@9.96.101.177:39180/]] > > arriving at [akka.tcp://flink@9.96.101.177:39180] inbound addresses > > are [akka.tcp://flink@9-96-101-177:39180] > > > > However, It is success when I use flink web-ui to submit the job. > > > > How to solve this problem? > > > > And otherwise, when I started the flink on yarn, the > > jobmanage.rpc.port and the web port both were changed to 39180 and 57724. > > The configuration following in flink-conf.yaml is just as default : > > > > jobmanager.rpc.port: 6123 > > > > and > > > > jobmanager.web.port: 8081 > > > > I started the flink on yarn using command : "./bin/yarn-session.sh -n 4". > > > > Why were the ports changed to 39180 and 57724? > > > > Many thanks if there is any help! > > > > HuangWHWHW > > 2017.1.13 > > > |
In reply to this post by Fabian Hueske-2
I found two problematic issues with Mesos HA mode which breaks it:
https://issues.apache.org/jira/browse/FLINK-5495 https://issues.apache.org/jira/browse/FLINK-5496 On Fri, Jan 13, 2017 at 11:29 AM, Fabian Hueske <[hidden email]> wrote: > I tested the Table API / SQL a bit. > > I implemented a windowed aggregation with the streaming Table API and it > produced the same results as a DataStream API implementation. > Joining a stream with a TableFunction also seemed to work well. > Moreover, I checked the results of a bunch of TPC-H queries (batch SQL) > and all produced correct results. > > > > 2017-01-12 17:45 GMT+01:00 Till Rohrmann <[hidden email]>: > >> I'm wondering whether we should not depend the webserver encryption on the >> global encryption activation and activating it instead per default. >> >> On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler <[hidden email]> >> wrote: >> >> > FLINK-5470 is a duplicate of FLINK-5298 for which there is also an open >> PR. >> > >> > FLINK-5472 is imo invalid since the webserver does support https, you >> just >> > have to enable it as per the security documentation. >> > >> > >> > On 12.01.2017 16:20, Till Rohrmann wrote: >> > >> > I also found an issue: >> > >> > https://issues.apache.org/jira/browse/FLINK-5470 >> > >> > I also noticed that Flink's webserver does not support https requests. >> It >> > might be worthwhile to add it, though. >> > >> > https://issues.apache.org/jira/browse/FLINK-5472 >> > >> > On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger <[hidden email]> >> > wrote: >> > >> >> I also found a bunch of issues >> >> >> >> https://issues.apache.org/jira/browse/FLINK-5465 >> >> https://issues.apache.org/jira/browse/FLINK-5462 >> >> https://issues.apache.org/jira/browse/FLINK-5464 >> >> https://issues.apache.org/jira/browse/FLINK-5463 >> >> >> >> >> >> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < <[hidden email]> >> >> [hidden email]> wrote: >> >> >> >> > I have another bugfix for 1.2.: >> >> > >> >> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) >> >> > >> >> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < <[hidden email]> >> >> [hidden email]>: >> >> > >> >> > > Hi, >> >> > > >> >> > > this depends a lot on the number of issues we find during the >> testing. >> >> > > >> >> > > >> >> > > These are the issues I found so far: >> >> > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) >> >> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) >> >> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) >> >> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) >> >> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui <[hidden email]> >> >> > wrote: >> >> > > >> >> > > > Do we have a probable time of 1.2 release? This month or Next >> month? >> >> > > > >> >> > > > -----邮件原件----- >> >> > > > 发件人: Robert Metzger [mailto: <[hidden email]> >> >> [hidden email]] >> >> > > > 发送时间: 2017年1月3日 20:44 >> >> > > > 收件人: [hidden email] >> >> > > > 抄送: [hidden email] >> >> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release >> >> > > candidate) >> >> > > > >> >> > > > Hi, >> >> > > > >> >> > > > First of all, I wish everybody a happy new year 2017. >> >> > > > >> >> > > > I've set user@flink in CC so that users who are interested in >> >> helping >> >> > > > with the testing get notified. Please respond only to the dev@ >> >> list to >> >> > > > keep the discussion there! >> >> > > > >> >> > > > According to the 1.2 release discussion thread, I've created a >> first >> >> > > > release candidate for Flink 1.2. >> >> > > > The release candidate will not be the final release, because I'm >> >> > certain >> >> > > > that we'll find at least one blocking issue in the candidate :) >> >> > > > >> >> > > > Therefore, the RC is meant as a testing only release candidate. >> >> > > > Please report every issue we need to fix before the next RC in >> this >> >> > > thread >> >> > > > so that we have a good overview. >> >> > > > >> >> > > > The release artifacts are located here: >> >> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ >> >> > > > >> >> > > > The maven staging repository is located here: >> >> > > > https://repository.apache.org/content/repositories/orgapache >> >> flink-1111 >> >> > > > >> >> > > > The release commit (in branch "release-1.2.0-rc0"): >> >> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced >> >> > > > >> >> > > > >> >> > > > Happy testing! >> >> > > > >> >> > > >> >> > >> >> >> > >> > >> > >> > > |
In reply to this post by Till Rohrmann
Hi Till,
Thanks very much ! It's helpful. HuangWHWHW 2017/1/16 -----邮件原件----- 发件人: Till Rohrmann [mailto:[hidden email]] 发送时间: 2017年1月15日 20:11 收件人: [hidden email] 抄送: [hidden email] 主题: Re: some questions about submit flink job on flink-yarn Hi Huang, the reason why you cannot use the IP address to send messages to your YARN JobManager is that we no longer resolve the hostname into an IP address. Instead we start the ActorSystem with the unresolved hostname. You can see this in the following log line: `Actor system bound to hostname 9-96-101-251`. Since Akka requires that the destination address of a message matches exactly the address to which the ActorSystem is bound, you have to use `9-96-101-251:38785`. This was recently changed. Concerning the ports, Flink chooses a random port for the `JobManager` in order to avoid port conflicts with other `JobManagers` running on the same node. With YARN you don't have control over where the `JobManager` is placed. However, you can use the configuration parameter `yarn.application-master.port` to specify a port or a port range for the application master/job manager. Additionally, the web frontends port is always overwritten and set to 0 which means random port selection when starting a yarn session. I hope this clarifies things a little bit. Cheers, Till On Sat, Jan 14, 2017 at 2:52 AM, huangwei (G) <[hidden email]> wrote: > Hi Till, > > The "9-96-101-177" is just the hostname. > I rerun the flink on yarn and here is the jobmanager.log, and sorry > for I blocked some sensitive log. By the way , the port(another > questions in my earlier mail) seems to be a random value(this time is 38785 and 35699). > I used flink-1.2.0, it works well on the yarn which is provided by > apache open source. But I run flink on a special yarn which was token > some safety reinforcement based on apache-yarn. I just have no idea about the ERROR log. > > Jobmanage.log: > > 2017-01-14 09:24:35,584 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - > ------------------------------------------------------------ > -------------------- > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Starting YARN ApplicationMaster / ResourceManager / > JobManager (Version: 1.2.0, Rev:82b1079, Date:04.01.2017 @ 17:38:23 > CST) > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Current user: admin > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle > Corporation - > 1.8/25.112-b15 > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Maximum heap size: 406 MiBytes > 2017-01-14 09:24:35,585 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - JAVA_HOME: /opt/huawei/Bigdata/jdk1.8.0_112/ > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Hadoop version: 2.7.2 > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - JVM Options: > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Xmx424M > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Dlog.file=/srv/BigData/hadoop/data1/nm/containerlogs/ > application_1483499303549_0043/container_1483499303549_ > 0043_01_000001/jobmanager.log > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Dlogback.configurationFile=file:logback.xml > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - -Dlog4j.configuration=file:log4j.properties > 2017-01-14 09:24:35,587 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Program Arguments: (none) > 2017-01-14 09:24:35,589 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - > ------------------------------------------------------------ > -------------------- > 2017-01-14 09:24:35,589 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Registered UNIX signal handlers for [TERM, HUP, INT] > 2017-01-14 09:24:35,591 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - remoteKeytabPrincipal obtained admin > 2017-01-14 09:24:35,592 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN daemon is running as: admin Yarn client user obtainer: > [hidden email] > 2017-01-14 09:24:35,596 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Loading config from directory > /srv/BigData/hadoop/data1/nm/ > localdir/usercache/admin/appcache/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.rpc.address, > 9.96.101.32 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.rpc.port, 6123 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.heap.mb, 256 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: taskmanager.heap.mb, 512 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: > taskmanager.numberOfTaskSlots, 1 > 2017-01-14 09:24:35,598 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: > taskmanager.memory.preallocate, false > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: parallelism.default, 1 > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: jobmanager.web.port, 8081 > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: security.keytab, > /home/demo/flink/release/flink-1.2.0/keytab/user.keytab > 2017-01-14 09:24:35,599 INFO org.apache.flink.configuration.GlobalConfiguration > - Loading configuration property: security.principal, admin > 2017-01-14 09:24:35,608 INFO org.apache.flink.runtime.security.JaasConfiguration > - Initializing JAAS configuration instance. Parameters: > /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/ > appcache/application_1483499303549_0043/container_ > 1483499303549_0043_01_000001/krb5.keytab, admin > 2017-01-14 09:24:35,609 INFO org.apache.flink.runtime.security.SecurityUtils > - SASL client auth for ZK will be disabled > 2017-01-14 09:24:35,824 INFO org.apache.hadoop.security.UserGroupInformation > - Login successful for user admin using keytab file > /srv/BigData/hadoop/data1/nm/localdir/usercache/admin/ > appcache/application_1483499303549_0043/container_ > 1483499303549_0043_01_000001/krb5.keytab > 2017-01-14 09:24:35,825 INFO org.apache.flink.runtime.security.SecurityUtils > - Hadoop user set to [hidden email] (auth:KERBEROS) > 2017-01-14 09:24:35,936 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN assigned hostname for application master: > 9-96-101-251 > 2017-01-14 09:24:35,936 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - keytabPath: /srv/BigData/hadoop/data1/nm/ > localdir/usercache/admin/appcache/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/krb5.keytab > 2017-01-14 09:24:35,938 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TaskManagers will be created with 1 task slots > 2017-01-14 09:24:35,938 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TaskManagers will be started with container size 1024 MB, > JVM heap size 424 MB, JVM direct memory limit 424 MB > 2017-01-14 09:24:35,943 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Trying to start actor system at 9.96.101.251:38785 > 2017-01-14 09:24:36,336 INFO akka.event.slf4j.Slf4jLogger > - Slf4jLogger started > 2017-01-14 09:24:36,438 INFO Remoting > - Starting remoting > 2017-01-14 09:24:36,547 INFO Remoting > - Remoting started; listening on addresses > :[akka.tcp://flink@9-96-101-251:38785] > 2017-01-14 09:24:36,551 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Actor system started at 9.96.101.251:38785 > 2017-01-14 09:24:36,551 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Actor system bound to hostname 9-96-101-251. > 2017-01-14 09:24:36,554 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Setting up resources for TaskManagers > 2017-01-14 09:24:36,554 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remoteKeytabPath obtained hdfs://hacluster/user/admin/. > flink/application_1483499303549_0043/user.keytab > 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remoteKeytabPrincipal obtained admin > 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remoteYarnConfPath obtained null > 2017-01-14 09:24:36,555 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - TM:remotekrb5Path obtained null > 2017-01-14 09:24:36,932 WARN org.apache.hadoop.util.NativeCodeLoader > - Unable to load native-hadoop library for your > platform... using builtin-java classes where applicable > 2017-01-14 09:24:36,945 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory > - The short-circuit local reads feature cannot be used because > libhadoop cannot be loaded. > 2017-01-14 09:24:36,949 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Adding keytab > hdfs://hacluster/user/admin/.flink/application_1483499303549_0043/user > .keytab to the AM container local resource bucket > 2017-01-14 09:24:37,085 INFO org.apache.flink.yarn.Utils > - Copying from file:/srv/BigData/hadoop/ > data1/nm/localdir/usercache/admin/appcache/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/ > e2943789-d80a-4abd-8ae9-2fc14cb1fd03-taskmanager-conf.yaml to > hdfs://hacluster/user/admin/.flink/application_ > 1483499303549_0043/e2943789-d80a-4abd-8ae9-2fc14cb1fd03- > taskmanager-conf.yaml > 2017-01-14 09:24:37,258 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Prepared local resource for modified yaml: resource { scheme: > "hdfs" host: "hacluster" port: -1 file: "/user/admin/.flink/ > application_1483499303549_0043/e2943789-d80a-4abd-8ae9- > 2fc14cb1fd03-taskmanager-conf.yaml" } size: 878 timestamp: > 1484357077250 > type: FILE visibility: APPLICATION > 2017-01-14 09:24:37,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Creating container launch context for TaskManagers > 2017-01-14 09:24:37,265 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Starting TaskManagers with command: $JAVA_HOME/bin/java > -Xms424m -Xmx424m -XX:MaxDirectMemorySize=424m > -Dlog.file=<LOG_DIR>/taskmanager.log > -Dlogback.configurationFile=file:./logback.xml > -Dlog4j.configuration=file:./log4j.properties > org.apache.flink.yarn.YarnTaskManager > --configDir . 1> <LOG_DIR>/taskmanager.out 2> > <LOG_DIR>/taskmanager.err > 2017-01-14 09:24:37,288 INFO org.apache.flink.runtime.blob.BlobServer > - Created BLOB server storage directory > /tmp/blobStore-f91bad88-1473-4e86-b151-f93dffa58baa > 2017-01-14 09:24:37,289 INFO org.apache.flink.runtime.blob.BlobServer > - Started BLOB server at 0.0.0.0:53972 - max > concurrent > requests: 50 - max backlog: 1000 > 2017-01-14 09:24:37,302 INFO org.apache.flink.runtime.metrics.MetricRegistry > - No metrics reporter configured, no metrics will be > exposed/reported. > 2017-01-14 09:24:37,307 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - Starting JobManager Web Frontend > 2017-01-14 09:24:37,310 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist > - Started memory archivist akka://flink/user/$a > 2017-01-14 09:24:37,311 INFO org.apache.flink.yarn.YarnJobManager > - Starting JobManager at akka.tcp://flink@9-96-101-251: > 38785/user/jobmanager. > 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils > - Determined location of JobManager log file: > /srv/BigData/hadoop/data1/nm/containerlogs/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.l > og > 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils > - Determined location of JobManager stdout file: > /srv/BigData/hadoop/data1/nm/containerlogs/application_ > 1483499303549_0043/container_1483499303549_0043_01_000001/jobmanager.o > ut > 2017-01-14 09:24:37,318 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Using directory > /tmp/flink-web-c4991b46-e637-4207-80ed-caef4cf5702e > for the web interface files > 2017-01-14 09:24:37,364 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Using directory > /tmp/flink-web-f8b7b6d7-b066-48a1-9536-40a3cdf42778 > for web frontend JAR file uploads > 2017-01-14 09:24:37,378 INFO org.apache.flink.yarn.YarnJobManager > - JobManager > akka.tcp://flink@9-96-101-251:38785/user/jobmanager > was granted leadership with leader session ID None. > 2017-01-14 09:24:37,560 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Web frontend listening at 0:0:0:0:0:0:0:0:35699 > 2017-01-14 09:24:37,561 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor > - Starting with JobManager > akka.tcp://flink@9-96-101-251:38785/user/jobmanager > on port 35699 > 2017-01-14 09:24:37,561 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever > - New leader reachable under akka://flink/user/jobmanager#- > 640052308:null. > 2017-01-14 09:24:37,568 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN application tolerates 4 failed TaskManager > containers before giving up > 2017-01-14 09:24:37,571 INFO org.apache.flink.yarn.YarnApplicationMasterRunner > - YARN Application Master started > 2017-01-14 09:24:37,579 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Initializing YARN resource master > 2017-01-14 09:24:37,605 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - > yarn.client.max-cached-nodemanagers-proxies > : 0 > 2017-01-14 09:24:37,606 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Registering Application Master with tracking url > http://9-96-101-251:35699 > 2017-01-14 09:24:37,641 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Trying to associate with JobManager leader > akka://flink/user/jobmanager#-640052308 > 2017-01-14 09:24:37,647 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Resource Manager associating with leading JobManager > Actor[akka://flink/user/jobmanager#-640052308] - leader session null > 2017-01-14 09:24:37,648 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 > megabytes memory. Pending requests: 1 > 2017-01-14 09:24:37,654 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 > megabytes memory. Pending requests: 2 > 2017-01-14 09:24:37,654 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 > megabytes memory. Pending requests: 3 > 2017-01-14 09:24:37,655 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Requesting new TaskManager container with 1024 > megabytes memory. Pending requests: 4 > 2017-01-14 09:24:38,681 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-96-101-177:26009 > 2017-01-14 09:24:38,692 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: > container_1483499303549_0043_01_000002 > - Remaining pending container requests: 3 > 2017-01-14 09:24:38,693 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357078692: Container: [ContainerId: > container_1483499303549_0043_01_000002, > NodeId: 9-96-101-177:26009, NodeHttpAddress: 9-96-101-177:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: > ContainerToken, > service: 9.96.101.177:26009 }, ] on host 9-96-101-177 > 2017-01-14 09:24:38,694 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-96-101-177:26009 > 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-96-101-251:26009 > 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-96-101-32:26009 > 2017-01-14 09:24:39,189 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl > - Received new token for : 9-91-8-160:26009 > 2017-01-14 09:24:39,190 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: > container_1483499303549_0043_01_000003 > - Remaining pending container requests: 2 > 2017-01-14 09:24:39,190 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357079190: Container: [ContainerId: > container_1483499303549_0043_01_000003, > NodeId: 9-96-101-251:26009, NodeHttpAddress: 9-96-101-251:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: > ContainerToken, > service: 9.96.101.251:26009 }, ] on host 9-96-101-251 > 2017-01-14 09:24:39,190 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-96-101-251:26009 > 2017-01-14 09:24:39,202 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: > container_1483499303549_0043_01_000004 > - Remaining pending container requests: 1 > 2017-01-14 09:24:39,202 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357079202: Container: [ContainerId: > container_1483499303549_0043_01_000004, > NodeId: 9-96-101-32:26009, NodeHttpAddress: 9-96-101-32:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: > ContainerToken, > service: 9.96.101.32:26009 }, ] on host 9-96-101-32 > 2017-01-14 09:24:39,202 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-96-101-32:26009 > 2017-01-14 09:24:39,217 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Received new container: > container_1483499303549_0043_01_000005 > - Remaining pending container requests: 0 > 2017-01-14 09:24:39,217 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - Launching TaskManager in container ContainerInLaunch @ > 1484357079217: Container: [ContainerId: > container_1483499303549_0043_01_000005, > NodeId: 9-91-8-160:26009, NodeHttpAddress: 9-91-8-160:26010, Resource: > <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: > ContainerToken, > service: 9.91.8.160:26009 }, ] on host 9-91-8-160 > 2017-01-14 09:24:39,217 INFO org.apache.hadoop.yarn.client.api.impl. > ContainerManagementProtocolProxy - Opening proxy : 9-91-8-160:26009 > 2017-01-14 09:24:43,348 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000003 has > started. > 2017-01-14 09:24:43,350 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-96-101-251 > (akka.tcp://flink@9-96-101-251:57010/user/taskmanager) as > 49800ab8cfcd1a11e45084a48281df75. Current number of registered hosts > is 1. Current number of alive task slots is 1. > 2017-01-14 09:24:44,881 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000002 has > started. > 2017-01-14 09:24:44,881 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-96-101-177 > (akka.tcp://flink@9-96-101-177:35778/user/taskmanager) as > 5d65baf1ec196cf3ac5bc43870156855. Current number of registered hosts > is 2. Current number of alive task slots is 2. > 2017-01-14 09:24:45,855 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000004 has > started. > 2017-01-14 09:24:45,855 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-96-101-32 > (akka.tcp://flink@9-96-101-32:58486/user/taskmanager) as > 80de4a47fa60536b78ea052cbadec7ee. Current number of registered hosts > is 3. Current number of alive task slots is 3. > 2017-01-14 09:24:46,018 INFO org.apache.flink.yarn.YarnFlinkResourceManager > - TaskManager container_1483499303549_0043_01_000005 has > started. > 2017-01-14 09:24:46,018 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at 9-91-8-160 > (akka.tcp://flink@9-91-8-160:47548/user/taskmanager) as > e9f2497d6223b2d704b3aced665a3c02. Current number of registered hosts > is 4. Current number of alive task slots is 4. > 2017-01-14 09:29:58,066 ERROR akka.remote.EndpointWriter > - dropping message [class > akka.actor.ActorSelectionMessage] for non-local recipient > [Actor[akka.tcp://flink@9.96.101.251:38785/]] > arriving at [akka.tcp://flink@9.96.101.251:38785] inbound addresses > are [akka.tcp://flink@9-96-101-251:38785] > > Thanks! > > HuangWHWHW > 2017/1/14 > > -----邮件原件----- > 发件人: Till Rohrmann [mailto:[hidden email]] > 发送时间: 2017年1月13日 18:22 > 收件人: [hidden email] > 抄送: [hidden email] > 主题: Re: some questions about submit flink job on flink-yarn > > Hi Huang, > > this seems to be very strange, because the JobManager’s actor system > has bound to the address 9-96-101-177 instead of 9.96.101.177. It > seems a if the . have been replaced by -. > > Could you maybe tell me which version of Flink you’re running and also > share the complete JobManager log with us? > > I tested it with the latest 1.2 SNAPSHOT version and there it seemed > to work. > > Cheers, > Till > > > On Fri, Jan 13, 2017 at 9:02 AM, huangwei (G) <[hidden email]> > wrote: > > > Dear All, > > > > I get an error in jobmanage.log following when I submit a flink job > > (batch/WordCount.jar) by using command : "./bin/flink run -m > > 9.96.101.177:39180 ./examples/batch/WordCount.jar". > > > > And the flink is on yarn cluster. > > > > Error in jobmanage.log : > > 2017-01-13 15:28:27,402 ERROR akka.remote.EndpointWriter > > - dropping message [class > > akka.actor.ActorSelectionMessage] for non-local recipient > > [Actor[akka.tcp://flink@9.96.101.177:39180/]] > > arriving at [akka.tcp://flink@9.96.101.177:39180] inbound addresses > > are [akka.tcp://flink@9-96-101-177:39180] > > > > However, It is success when I use flink web-ui to submit the job. > > > > How to solve this problem? > > > > And otherwise, when I started the flink on yarn, the > > jobmanage.rpc.port and the web port both were changed to 39180 and 57724. > > The configuration following in flink-conf.yaml is just as default : > > > > jobmanager.rpc.port: 6123 > > > > and > > > > jobmanager.web.port: 8081 > > > > I started the flink on yarn using command : "./bin/yarn-session.sh -n 4". > > > > Why were the ports changed to 39180 and 57724? > > > > Many thanks if there is any help! > > > > HuangWHWHW > > 2017.1.13 > > > |
In reply to this post by Till Rohrmann
A user reported that outer joins on the Table API and SQL compute wrong
results: https://issues.apache.org/jira/browse/FLINK-5498 2017-01-15 20:23 GMT+01:00 Till Rohrmann <[hidden email]>: > I found two problematic issues with Mesos HA mode which breaks it: > > https://issues.apache.org/jira/browse/FLINK-5495 > https://issues.apache.org/jira/browse/FLINK-5496 > > On Fri, Jan 13, 2017 at 11:29 AM, Fabian Hueske <[hidden email]> wrote: > > > I tested the Table API / SQL a bit. > > > > I implemented a windowed aggregation with the streaming Table API and it > > produced the same results as a DataStream API implementation. > > Joining a stream with a TableFunction also seemed to work well. > > Moreover, I checked the results of a bunch of TPC-H queries (batch SQL) > > and all produced correct results. > > > > > > > > 2017-01-12 17:45 GMT+01:00 Till Rohrmann <[hidden email]>: > > > >> I'm wondering whether we should not depend the webserver encryption on > the > >> global encryption activation and activating it instead per default. > >> > >> On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler <[hidden email]> > >> wrote: > >> > >> > FLINK-5470 is a duplicate of FLINK-5298 for which there is also an > open > >> PR. > >> > > >> > FLINK-5472 is imo invalid since the webserver does support https, you > >> just > >> > have to enable it as per the security documentation. > >> > > >> > > >> > On 12.01.2017 16:20, Till Rohrmann wrote: > >> > > >> > I also found an issue: > >> > > >> > https://issues.apache.org/jira/browse/FLINK-5470 > >> > > >> > I also noticed that Flink's webserver does not support https requests. > >> It > >> > might be worthwhile to add it, though. > >> > > >> > https://issues.apache.org/jira/browse/FLINK-5472 > >> > > >> > On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger <[hidden email] > > > >> > wrote: > >> > > >> >> I also found a bunch of issues > >> >> > >> >> https://issues.apache.org/jira/browse/FLINK-5465 > >> >> https://issues.apache.org/jira/browse/FLINK-5462 > >> >> https://issues.apache.org/jira/browse/FLINK-5464 > >> >> https://issues.apache.org/jira/browse/FLINK-5463 > >> >> > >> >> > >> >> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < <[hidden email]> > >> >> [hidden email]> wrote: > >> >> > >> >> > I have another bugfix for 1.2.: > >> >> > > >> >> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) > >> >> > > >> >> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < <[hidden email]> > >> >> [hidden email]>: > >> >> > > >> >> > > Hi, > >> >> > > > >> >> > > this depends a lot on the number of issues we find during the > >> testing. > >> >> > > > >> >> > > > >> >> > > These are the issues I found so far: > >> >> > > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) > >> >> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) > >> >> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) > >> >> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) > >> >> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui < > [hidden email]> > >> >> > wrote: > >> >> > > > >> >> > > > Do we have a probable time of 1.2 release? This month or Next > >> month? > >> >> > > > > >> >> > > > -----邮件原件----- > >> >> > > > 发件人: Robert Metzger [mailto: <[hidden email]> > >> >> [hidden email]] > >> >> > > > 发送时间: 2017年1月3日 20:44 > >> >> > > > 收件人: [hidden email] > >> >> > > > 抄送: [hidden email] > >> >> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing > release > >> >> > > candidate) > >> >> > > > > >> >> > > > Hi, > >> >> > > > > >> >> > > > First of all, I wish everybody a happy new year 2017. > >> >> > > > > >> >> > > > I've set user@flink in CC so that users who are interested in > >> >> helping > >> >> > > > with the testing get notified. Please respond only to the dev@ > >> >> list to > >> >> > > > keep the discussion there! > >> >> > > > > >> >> > > > According to the 1.2 release discussion thread, I've created a > >> first > >> >> > > > release candidate for Flink 1.2. > >> >> > > > The release candidate will not be the final release, because > I'm > >> >> > certain > >> >> > > > that we'll find at least one blocking issue in the candidate :) > >> >> > > > > >> >> > > > Therefore, the RC is meant as a testing only release candidate. > >> >> > > > Please report every issue we need to fix before the next RC in > >> this > >> >> > > thread > >> >> > > > so that we have a good overview. > >> >> > > > > >> >> > > > The release artifacts are located here: > >> >> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > >> >> > > > > >> >> > > > The maven staging repository is located here: > >> >> > > > https://repository.apache.org/content/repositories/orgapache > >> >> flink-1111 > >> >> > > > > >> >> > > > The release commit (in branch "release-1.2.0-rc0"): > >> >> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > >> >> > > > > >> >> > > > > >> >> > > > Happy testing! > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > > >> > > >> > > >> > > > > > |
Hi all,
I would like to do a proper voting RC1 early this week. From the issues mentioned here, most of them have pull requests or were changed to a lower priority. Once we've merged all outstanding PRs, I'll create the next RC. Regards, Robert On Mon, Jan 16, 2017 at 12:13 PM, Fabian Hueske <[hidden email]> wrote: > A user reported that outer joins on the Table API and SQL compute wrong > results: > > https://issues.apache.org/jira/browse/FLINK-5498 > > 2017-01-15 20:23 GMT+01:00 Till Rohrmann <[hidden email]>: > > > I found two problematic issues with Mesos HA mode which breaks it: > > > > https://issues.apache.org/jira/browse/FLINK-5495 > > https://issues.apache.org/jira/browse/FLINK-5496 > > > > On Fri, Jan 13, 2017 at 11:29 AM, Fabian Hueske <[hidden email]> > wrote: > > > > > I tested the Table API / SQL a bit. > > > > > > I implemented a windowed aggregation with the streaming Table API and > it > > > produced the same results as a DataStream API implementation. > > > Joining a stream with a TableFunction also seemed to work well. > > > Moreover, I checked the results of a bunch of TPC-H queries (batch SQL) > > > and all produced correct results. > > > > > > > > > > > > 2017-01-12 17:45 GMT+01:00 Till Rohrmann <[hidden email]>: > > > > > >> I'm wondering whether we should not depend the webserver encryption on > > the > > >> global encryption activation and activating it instead per default. > > >> > > >> On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler <[hidden email] > > > > >> wrote: > > >> > > >> > FLINK-5470 is a duplicate of FLINK-5298 for which there is also an > > open > > >> PR. > > >> > > > >> > FLINK-5472 is imo invalid since the webserver does support https, > you > > >> just > > >> > have to enable it as per the security documentation. > > >> > > > >> > > > >> > On 12.01.2017 16:20, Till Rohrmann wrote: > > >> > > > >> > I also found an issue: > > >> > > > >> > https://issues.apache.org/jira/browse/FLINK-5470 > > >> > > > >> > I also noticed that Flink's webserver does not support https > requests. > > >> It > > >> > might be worthwhile to add it, though. > > >> > > > >> > https://issues.apache.org/jira/browse/FLINK-5472 > > >> > > > >> > On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger < > [hidden email] > > > > > >> > wrote: > > >> > > > >> >> I also found a bunch of issues > > >> >> > > >> >> https://issues.apache.org/jira/browse/FLINK-5465 > > >> >> https://issues.apache.org/jira/browse/FLINK-5462 > > >> >> https://issues.apache.org/jira/browse/FLINK-5464 > > >> >> https://issues.apache.org/jira/browse/FLINK-5463 > > >> >> > > >> >> > > >> >> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < < > [hidden email]> > > >> >> [hidden email]> wrote: > > >> >> > > >> >> > I have another bugfix for 1.2.: > > >> >> > > > >> >> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) > > >> >> > > > >> >> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < <[hidden email] > > > > >> >> [hidden email]>: > > >> >> > > > >> >> > > Hi, > > >> >> > > > > >> >> > > this depends a lot on the number of issues we find during the > > >> testing. > > >> >> > > > > >> >> > > > > >> >> > > These are the issues I found so far: > > >> >> > > > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) > > >> >> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui < > > [hidden email]> > > >> >> > wrote: > > >> >> > > > > >> >> > > > Do we have a probable time of 1.2 release? This month or Next > > >> month? > > >> >> > > > > > >> >> > > > -----邮件原件----- > > >> >> > > > 发件人: Robert Metzger [mailto: <[hidden email]> > > >> >> [hidden email]] > > >> >> > > > 发送时间: 2017年1月3日 20:44 > > >> >> > > > 收件人: [hidden email] > > >> >> > > > 抄送: [hidden email] > > >> >> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing > > release > > >> >> > > candidate) > > >> >> > > > > > >> >> > > > Hi, > > >> >> > > > > > >> >> > > > First of all, I wish everybody a happy new year 2017. > > >> >> > > > > > >> >> > > > I've set user@flink in CC so that users who are interested > in > > >> >> helping > > >> >> > > > with the testing get notified. Please respond only to the > dev@ > > >> >> list to > > >> >> > > > keep the discussion there! > > >> >> > > > > > >> >> > > > According to the 1.2 release discussion thread, I've created > a > > >> first > > >> >> > > > release candidate for Flink 1.2. > > >> >> > > > The release candidate will not be the final release, because > > I'm > > >> >> > certain > > >> >> > > > that we'll find at least one blocking issue in the candidate > :) > > >> >> > > > > > >> >> > > > Therefore, the RC is meant as a testing only release > candidate. > > >> >> > > > Please report every issue we need to fix before the next RC > in > > >> this > > >> >> > > thread > > >> >> > > > so that we have a good overview. > > >> >> > > > > > >> >> > > > The release artifacts are located here: > > >> >> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ > > >> >> > > > > > >> >> > > > The maven staging repository is located here: > > >> >> > > > https://repository.apache.org/content/repositories/orgapache > > >> >> flink-1111 > > >> >> > > > > > >> >> > > > The release commit (in branch "release-1.2.0-rc0"): > > >> >> > > > http://git-wip-us.apache.org/repos/asf/flink/commit/f3c59ced > > >> >> > > > > > >> >> > > > > > >> >> > > > Happy testing! > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > >> > > > >> > > > >> > > > > > > > > > |
RC1 creation is in progress ...
On Mon, Jan 23, 2017 at 10:33 AM, Robert Metzger <[hidden email]> wrote: > Hi all, > > I would like to do a proper voting RC1 early this week. > From the issues mentioned here, most of them have pull requests or were > changed to a lower priority. > Once we've merged all outstanding PRs, I'll create the next RC. > > Regards, > Robert > > > On Mon, Jan 16, 2017 at 12:13 PM, Fabian Hueske <[hidden email]> wrote: > >> A user reported that outer joins on the Table API and SQL compute wrong >> results: >> >> https://issues.apache.org/jira/browse/FLINK-5498 >> >> 2017-01-15 20:23 GMT+01:00 Till Rohrmann <[hidden email]>: >> >> > I found two problematic issues with Mesos HA mode which breaks it: >> > >> > https://issues.apache.org/jira/browse/FLINK-5495 >> > https://issues.apache.org/jira/browse/FLINK-5496 >> > >> > On Fri, Jan 13, 2017 at 11:29 AM, Fabian Hueske <[hidden email]> >> wrote: >> > >> > > I tested the Table API / SQL a bit. >> > > >> > > I implemented a windowed aggregation with the streaming Table API and >> it >> > > produced the same results as a DataStream API implementation. >> > > Joining a stream with a TableFunction also seemed to work well. >> > > Moreover, I checked the results of a bunch of TPC-H queries (batch >> SQL) >> > > and all produced correct results. >> > > >> > > >> > > >> > > 2017-01-12 17:45 GMT+01:00 Till Rohrmann <[hidden email]>: >> > > >> > >> I'm wondering whether we should not depend the webserver encryption >> on >> > the >> > >> global encryption activation and activating it instead per default. >> > >> >> > >> On Thu, Jan 12, 2017 at 4:54 PM, Chesnay Schepler < >> [hidden email]> >> > >> wrote: >> > >> >> > >> > FLINK-5470 is a duplicate of FLINK-5298 for which there is also an >> > open >> > >> PR. >> > >> > >> > >> > FLINK-5472 is imo invalid since the webserver does support https, >> you >> > >> just >> > >> > have to enable it as per the security documentation. >> > >> > >> > >> > >> > >> > On 12.01.2017 16:20, Till Rohrmann wrote: >> > >> > >> > >> > I also found an issue: >> > >> > >> > >> > https://issues.apache.org/jira/browse/FLINK-5470 >> > >> > >> > >> > I also noticed that Flink's webserver does not support https >> requests. >> > >> It >> > >> > might be worthwhile to add it, though. >> > >> > >> > >> > https://issues.apache.org/jira/browse/FLINK-5472 >> > >> > >> > >> > On Thu, Jan 12, 2017 at 11:24 AM, Robert Metzger < >> [hidden email] >> > > >> > >> > wrote: >> > >> > >> > >> >> I also found a bunch of issues >> > >> >> >> > >> >> https://issues.apache.org/jira/browse/FLINK-5465 >> > >> >> https://issues.apache.org/jira/browse/FLINK-5462 >> > >> >> https://issues.apache.org/jira/browse/FLINK-5464 >> > >> >> https://issues.apache.org/jira/browse/FLINK-5463 >> > >> >> >> > >> >> >> > >> >> On Thu, Jan 12, 2017 at 9:56 AM, Fabian Hueske < < >> [hidden email]> >> > >> >> [hidden email]> wrote: >> > >> >> >> > >> >> > I have another bugfix for 1.2.: >> > >> >> > >> > >> >> > https://issues.apache.org/jira/browse/FLINK-2662 (pending PR) >> > >> >> > >> > >> >> > 2017-01-10 15:16 GMT+01:00 Robert Metzger < < >> [hidden email]> >> > >> >> [hidden email]>: >> > >> >> > >> > >> >> > > Hi, >> > >> >> > > >> > >> >> > > this depends a lot on the number of issues we find during the >> > >> testing. >> > >> >> > > >> > >> >> > > >> > >> >> > > These are the issues I found so far: >> > >> >> > > >> > >> >> > > https://issues.apache.org/jira/browse/FLINK-5379 (unresolved) >> > >> >> > > https://issues.apache.org/jira/browse/FLINK-5383 (resolved) >> > >> >> > > https://issues.apache.org/jira/browse/FLINK-5382 (resolved) >> > >> >> > > https://issues.apache.org/jira/browse/FLINK-5381 (resolved) >> > >> >> > > https://issues.apache.org/jira/browse/FLINK-5380 (pending PR) >> > >> >> > > >> > >> >> > > >> > >> >> > > >> > >> >> > > >> > >> >> > > On Tue, Jan 10, 2017 at 11:58 AM, shijinkui < >> > [hidden email]> >> > >> >> > wrote: >> > >> >> > > >> > >> >> > > > Do we have a probable time of 1.2 release? This month or >> Next >> > >> month? >> > >> >> > > > >> > >> >> > > > -----邮件原件----- >> > >> >> > > > 发件人: Robert Metzger [mailto: <[hidden email]> >> > >> >> [hidden email]] >> > >> >> > > > 发送时间: 2017年1月3日 20:44 >> > >> >> > > > 收件人: [hidden email] >> > >> >> > > > 抄送: [hidden email] >> > >> >> > > > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing >> > release >> > >> >> > > candidate) >> > >> >> > > > >> > >> >> > > > Hi, >> > >> >> > > > >> > >> >> > > > First of all, I wish everybody a happy new year 2017. >> > >> >> > > > >> > >> >> > > > I've set user@flink in CC so that users who are interested >> in >> > >> >> helping >> > >> >> > > > with the testing get notified. Please respond only to the >> dev@ >> > >> >> list to >> > >> >> > > > keep the discussion there! >> > >> >> > > > >> > >> >> > > > According to the 1.2 release discussion thread, I've >> created a >> > >> first >> > >> >> > > > release candidate for Flink 1.2. >> > >> >> > > > The release candidate will not be the final release, because >> > I'm >> > >> >> > certain >> > >> >> > > > that we'll find at least one blocking issue in the >> candidate :) >> > >> >> > > > >> > >> >> > > > Therefore, the RC is meant as a testing only release >> candidate. >> > >> >> > > > Please report every issue we need to fix before the next RC >> in >> > >> this >> > >> >> > > thread >> > >> >> > > > so that we have a good overview. >> > >> >> > > > >> > >> >> > > > The release artifacts are located here: >> > >> >> > > > http://people.apache.org/~rmetzger/flink-1.2.0-rc0/ >> > >> >> > > > >> > >> >> > > > The maven staging repository is located here: >> > >> >> > > > https://repository.apache.org/ >> content/repositories/orgapache >> > >> >> flink-1111 >> > >> >> > > > >> > >> >> > > > The release commit (in branch "release-1.2.0-rc0"): >> > >> >> > > > http://git-wip-us.apache.org/r >> epos/asf/flink/commit/f3c59ced >> > >> >> > > > >> > >> >> > > > >> > >> >> > > > Happy testing! >> > >> >> > > > >> > >> >> > > >> > >> >> > >> > >> >> >> > >> > >> > >> > >> > >> > >> > >> >> > > >> > > >> > >> > > |
Free forum by Nabble | Edit this page |