Hi,
I have written a job that reads a SequenceFile from HDFS using the Hadoop-Compatibility add-on. Doing so results in a TimeoutException. I’m using flink-0.9-SNAPSHOT with PR 342 ( https://github.com/apache/flink/pull/342 ). Furthermore I’m running flink on yarn with two TM using flink-yarn-0.9-SNAPSHOT/bin/yarn-session.sh -n 2. Is this a bug or is there something wrong with the configuration? 01/28/2015 11:42:52 Job execution switched to status RUNNING. 01/28/2015 11:42:52 CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to SCHEDULED 01/28/2015 11:42:52 CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to DEPLOYING 01/28/2015 11:42:52 CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to RUNNING 01/28/2015 11:44:32 CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to FAILED java.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more 01/28/2015 11:44:32 Job execution switched to status FAILING. 01/28/2015 11:44:32 GroupReduce (GroupReduce at main(ThiaziParser.java:40))(1/1) switched to CANCELED 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs://cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/authors) - UTF-8)(1/1) switched to CANCELED 01/28/2015 11:44:32 CHAIN GroupReduce (GroupReduce at main(ThiaziParser.java:74)) -> Filter (Filter at main(ThiaziParser.java:97))(1/1) switched to CANCELED 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs://cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/posts) - UTF-8)(1/1) switched to CANCELED 01/28/2015 11:44:32 CHAIN FlatMap (FlatMap at main(ThiaziParser.java:126)) -> Combine(SUM(1), at main(ThiaziParser.java:140)(1/1) switched to CANCELED 01/28/2015 11:44:32 Reduce (SUM(1), at main(ThiaziParser.java:140)(1/1) switched to CANCELED 01/28/2015 11:44:32 DataSink(CsvOutputFormat (path: hdfs://cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/wordcount, delimiter: ,))(1/1) switched to CANCELED 01/28/2015 11:44:32 GroupReduce (GroupReduce at main(ThiaziParser.java:106))(1/1) switched to CANCELED 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs://cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/threads) - UTF-8)(1/1) switched to CANCELED 01/28/2015 11:44:32 Job execution switched to status FAILED. Error: The program execution failed: java.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more |
Hi,
it seems that you are not subscribed to our mailing list, so I had to manually accept your mail. Would be good if you could subscribe. Can you send us also the log output of the JobManager? If your YARN cluster has log aggregation activated, you can retrieve the logs of a stopped YARN session using: yarn logs -applicationId <AppId> watch out for the jobmanager-main.log or so file. I suspect that there has been an exception on the JobManager. Best, Robert On Wed, Jan 28, 2015 at 12:01 PM, Bruecke, Christoph < [hidden email]> wrote: > Hi, > > I have written a job that reads a SequenceFile from HDFS using the > Hadoop-Compatibility add-on. Doing so results in a TimeoutException. I’m > using flink-0.9-SNAPSHOT with PR 342 ( > https://github.com/apache/flink/pull/342 ). Furthermore I’m running flink > on yarn with two TM using flink-yarn-0.9-SNAPSHOT/bin/yarn-session.sh -n 2. > > Is this a bug or is there something wrong with the configuration? > > 01/28/2015 11:42:52 Job execution switched to status RUNNING. > 01/28/2015 11:42:52 CHAIN DataSource (at > createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to SCHEDULED > 01/28/2015 11:42:52 CHAIN DataSource (at > createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to DEPLOYING > 01/28/2015 11:42:52 CHAIN DataSource (at > createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to RUNNING > 01/28/2015 11:44:32 CHAIN DataSource (at > createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to FAILED > java.lang.RuntimeException: Requesting the next InputSplit failed. > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [100 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > > 01/28/2015 11:44:32 Job execution switched to status FAILING. > 01/28/2015 11:44:32 GroupReduce (GroupReduce at > main(ThiaziParser.java:40))(1/1) switched to CANCELED > 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/authors) > - UTF-8)(1/1) switched to CANCELED > 01/28/2015 11:44:32 CHAIN GroupReduce (GroupReduce at > main(ThiaziParser.java:74)) -> Filter (Filter at > main(ThiaziParser.java:97))(1/1) switched to CANCELED > 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/posts) - > UTF-8)(1/1) switched to CANCELED > 01/28/2015 11:44:32 CHAIN FlatMap (FlatMap at > main(ThiaziParser.java:126)) -> Combine(SUM(1), at > main(ThiaziParser.java:140)(1/1) switched to CANCELED > 01/28/2015 11:44:32 Reduce (SUM(1), at > main(ThiaziParser.java:140)(1/1) switched to CANCELED > 01/28/2015 11:44:32 DataSink(CsvOutputFormat (path: hdfs:// > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/wordcount, > delimiter: ,))(1/1) switched to CANCELED > 01/28/2015 11:44:32 GroupReduce (GroupReduce at > main(ThiaziParser.java:106))(1/1) switched to CANCELED > 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/threads) > - UTF-8)(1/1) switched to CANCELED > 01/28/2015 11:44:32 Job execution switched to status FAILED. > Error: The program execution failed: java.lang.RuntimeException: > Requesting the next InputSplit failed. > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [100 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > > > |
Hi Robert,
thanks for the quick response. Here is the jobmanager-main.log: PS: I’m subscribed now. 11:09:16,144 INFO org.apache.flink.yarn.ApplicationMaster$ - YARN daemon runs as hadoop setting user to execute Flink ApplicationMaster/JobManager to hadoop 11:09:16,199 INFO org.apache.flink.yarn.Utils - Found YARN_CONF_DIR, adding it to configuration 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ - Start job manager for yarn 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ - Config path: /data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001. 11:09:17,179 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 11:09:17,271 INFO Remoting - Starting remoting 11:09:17,556 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://[hidden email]:42643] 11:09:17,569 INFO org.apache.flink.yarn.ApplicationMaster$ - Start job manager actor. 11:09:17,570 INFO org.apache.flink.yarn.ApplicationMaster$ - Generate configuration file for application master. 11:09:17,579 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Starting job manager at akka://flink/user/jobmanager. 11:09:17,590 INFO org.apache.flink.yarn.ApplicationMaster$ - Start yarn session on job manager. 11:09:17,591 INFO org.apache.flink.yarn.ApplicationMaster$ - Application Master properly initiated. Await termination of actor system. 11:09:17,601 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server on port 35605 11:09:17,616 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Started job manager. Waiting for incoming messages. 11:09:17,621 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer - Setting up web info server, using web-root directoryjar:file:/data/4/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/filecache/11/flink-dist-0.9-SNAPSHOT-yarn-uberjar.jar!/web-docs-infoserver. 11:09:17,621 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer - Web info server will display information about flink job-manager on localhost, port 8081. 11:09:17,733 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer - Starting web info server for JobManager on port 8081 11:09:17,734 INFO org.eclipse.jetty.util.log - jetty-0.9-SNAPSHOT 11:09:17,770 INFO org.eclipse.jetty.util.log - Started SelectChannelConnector@0.0.0.0:8081 11:09:17,784 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Start yarn session. 11:09:17,784 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Requesting 2 task managers. 11:09:18,414 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at cloud-11/130.149.21.15:8030 11:09:18,423 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-nodemanagers-proxies : 500 11:09:18,425 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Registering ApplicationMaster with tracking url http://cloud-29.dima.tu-berlin.de:8081. 11:09:18,787 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Requesting TaskManager container 0. 11:09:18,794 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Requesting TaskManager container 1. 11:09:18,903 INFO org.apache.flink.yarn.Utils - Copying from file:/data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001/flink-conf-modified.yaml to hdfs://cloud-11.dima.tu-berlin.de:60010/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml 11:09:19,280 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "cloud-11.dima.tu-berlin.de" port: 60010 file: "/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml" } size: 3121 timestamp: 1422439759181 type: FILE visibility: APPLICATION 11:09:19,286 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Create container launch context. 11:09:19,300 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Starting TM with command=$JAVA_HOME/bin/java -Xmx819m -Dlog.file="<LOG_DIR>/taskmanager.log" -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.appMaster.YarnTaskManagerRunner --configDir . 1> <LOG_DIR>/taskmanager-stdout.log 2> <LOG_DIR>/taskmanager-stderr.log 11:09:19,619 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Got new container for TM container_1420727594991_0068_01_000002 on host cloud-26.dima.tu-berlin.de 11:09:19,621 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Launching container #1. 11:09:19,622 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : cloud-26.dima.tu-berlin.de:8045 11:09:19,828 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Got new container for TM container_1420727594991_0068_01_000003 on host cloud-31.dima.tu-berlin.de 11:09:19,829 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Launching container #2. 11:09:19,831 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : cloud-31.dima.tu-berlin.de:8045 11:09:25,748 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at akka.tcp://[hidden email]:51449/user/taskmanager as ca731dbbfe941a63077c9c148254607c. Current number of registered hosts is 1. 11:09:25,845 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at akka.tcp://[hidden email]:43200/user/taskmanager as 7bf3aa638e00682239c9d8460d142191. Current number of registered hosts is 2. 11:11:13,837 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://CliFrontendActorSystem@130.149.21.15:50292] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 11:14:30,892 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Received job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015). 11:14:30,997 INFO org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.interval is deprecated. Instead, use mapreduce.job.end-notification.retry.interval 11:14:30,998 INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 11:14:30,999 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use mapreduce.jobtracker.retiredjobs.cache.size 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.reduces is deprecated. Instead, use mapreduce.task.profile.reduces 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use mapreduce.job.jvm.numtasks 11:14:31,001 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 11:14:31,003 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.http.address is deprecated. Instead, use mapreduce.tasktracker.http.address 11:14:31,004 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir 11:14:31,005 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.report.address is deprecated. Instead, use mapreduce.tasktracker.report.address 11:14:31,006 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.healthChecker.interval is deprecated. Instead, use mapreduce.tasktracker.healthchecker.interval 11:14:31,007 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.child.tmp is deprecated. Instead, use mapreduce.task.tmp.dir 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.taskmemorymanager.monitoring-interval is deprecated. Instead, use mapreduce.tasktracker.taskmemorymanager.monitoringinterval 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.shuffle.connect.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.connect.timeout 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.speculativeCap is deprecated. Instead, use mapreduce.job.speculative.speculativecap 11:14:31,009 INFO org.apache.hadoop.conf.Configuration.deprecation - io.sort.spill.percent is deprecated. Instead, use mapreduce.map.sort.spill.percent 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.shuffle.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.shuffle.input.buffer.percent 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.map.max.skip.records is deprecated. Instead, use mapreduce.map.skip.maxrecords 11:14:31,011 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.maps is deprecated. Instead, use mapreduce.task.profile.maps 11:14:31,012 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.merge.recordsBeforeProgress is deprecated. Instead, use mapreduce.task.merge.progress.records 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.retain.hours is deprecated. Instead, use mapreduce.job.userlog.retain.hours 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use mapreduce.job.speculative.slownodethreshold 11:14:31,015 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, use mapreduce.tasktracker.reduce.tasks.maximum 11:14:31,017 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level 11:14:31,018 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.restart.recover is deprecated. Instead, use mapreduce.jobtracker.restart.recover 11:14:31,020 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.inmem.merge.threshold is deprecated. Instead, use mapreduce.reduce.merge.inmem.threshold 11:14:31,021 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.acls.enabled is deprecated. Instead, use mapreduce.cluster.acls.enabled 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.dns.nameserver is deprecated. Instead, use mapreduce.tasktracker.dns.nameserver 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 11:14:31,025 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.shuffle.merge.percent is deprecated. Instead, use mapreduce.reduce.shuffle.merge.percent 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.lru.cache.size 11:14:31,029 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename 11:14:31,030 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.hours 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use mapreduce.tasktracker.map.tasks.maximum 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation - jobclient.completion.poll.interval is deprecated. Instead, use mapreduce.client.completion.pollinterval 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.dir 11:14:31,033 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.slowstart.completed.maps is deprecated. Instead, use mapreduce.job.reduce.slowstart.completedmaps 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation - dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode 11:14:31,037 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.instrumentation is deprecated. Instead, use mapreduce.jobtracker.instrumentation 11:14:31,038 INFO org.apache.hadoop.conf.Configuration.deprecation - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.attempts.to.start.skipping is deprecated. Instead, use mapreduce.task.skip.start.attempts 11:14:31,040 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.task-controller is deprecated. Instead, use mapreduce.tasktracker.taskcontroller 11:14:31,041 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.limit.kb is deprecated. Instead, use mapreduce.task.userlog.limit.kb 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir.minspacekill is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacekill 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.jobtracker.split.metainfo.maxsize is deprecated. Instead, use mapreduce.job.split.metainfo.maxsize 11:14:31,045 INFO org.apache.hadoop.conf.Configuration.deprecation - jobclient.progress.monitor.poll.interval is deprecated. Instead, use mapreduce.client.progressmonitor.pollinterval 11:14:31,046 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 11:14:31,047 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile is deprecated. Instead, use mapreduce.task.profile 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.parallel.copies is deprecated. Instead, use mapreduce.reduce.shuffle.parallelcopies 11:14:31,049 INFO org.apache.hadoop.conf.Configuration.deprecation - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout 11:14:31,052 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.heartbeats.in.second is deprecated. Instead, use mapreduce.jobtracker.heartbeats.in.second 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.cache.levels is deprecated. Instead, use mapreduce.jobtracker.taskcache.levels 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.output.compression.codec is deprecated. Instead, use mapreduce.map.output.compress.codec 11:14:31,054 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.dns.interface is deprecated. Instead, use mapreduce.tasktracker.dns.interface 11:14:31,055 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.max.tracker.failures is deprecated. Instead, use mapreduce.job.maxtaskfailures.per.tracker 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation - dfs.df.interval is deprecated. Instead, use fs.df.interval 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead, use mapreduce.tasktracker.tasks.sleeptimebeforesigkill 11:14:31,057 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.max.tracker.blacklists is deprecated. Instead, use mapreduce.jobtracker.tasktracker.maxblacklists 11:14:31,058 INFO org.apache.hadoop.conf.Configuration.deprecation - jobclient.output.filter is deprecated. Instead, use mapreduce.client.output.filter 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.attempts is deprecated. Instead, use mapreduce.job.end-notification.retry.attempts 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.taskScheduler is deprecated. Instead, use mapreduce.jobtracker.taskscheduler 11:14:31,060 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.slowTaskThreshold is deprecated. Instead, use mapreduce.job.speculative.slowtaskthreshold 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.indexcache.mb is deprecated. Instead, use mapreduce.tasktracker.indexcache.mb 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation - tasktracker.http.threads is deprecated. Instead, use mapreduce.tasktracker.http.threads 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.handler.count is deprecated. Instead, use mapreduce.jobtracker.handler.count 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation - keep.failed.task.files is deprecated. Instead, use mapreduce.task.files.preserve.failedtasks 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.job.history.block.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.block.size 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.reduce.max.skip.groups is deprecated. Instead, use mapreduce.reduce.skip.maxgroups 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir.minspacestart is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacestart 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use mapreduce.jobtracker.maxtasks.perjob 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.max.attempts is deprecated. Instead, use mapreduce.reduce.maxattempts 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.shuffle.read.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.read.timeout 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.instrumentation is deprecated. Instead, use mapreduce.tasktracker.instrumentation 11:14:31,070 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.expiry.interval is deprecated. Instead, use mapreduce.jobtracker.expire.trackers.interval 11:14:31,071 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.active 11:14:31,074 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation - hadoop.native.lib is deprecated. Instead, use io.native.lib.available 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 11:14:31,076 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.input.buffer.percent 11:14:31,149 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 11:14:31,177 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Scheduling job Flink Java Job at Wed Jan 28 11:14:28 CET 2015. 11:14:31,188 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to cloud-26 11:14:31,202 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015) changed to RUNNING. 11:14:32,525 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host cloud-26 11:14:32,603 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[hidden email]:51449] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 11:16:12,574 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015) changed to FAILINGjava.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more . 11:16:12,584 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015) changed to FAILEDjava.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more . 11:17:53,055 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Received job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015). 11:17:53,258 INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 11:17:53,312 INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 11:17:53,320 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 11:17:53,347 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 11:17:53,352 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to cloud-31 11:17:53,353 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Scheduling job Flink Java Job at Wed Jan 28 11:17:50 CET 2015. 11:17:53,354 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015) changed to RUNNING. 11:17:54,646 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host cloud-31 11:17:54,715 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[hidden email]:43200] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 11:19:34,684 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more . 11:19:34,697 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more . 11:42:52,441 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Received job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015). 11:42:52,650 INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 11:42:52,705 INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 11:42:52,713 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 11:42:52,740 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 11:42:52,747 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Scheduling job Flink Java Job at Wed Jan 28 11:42:50 CET 2015. 11:42:52,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to cloud-26 11:42:52,750 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015) changed to RUNNING. 11:42:52,956 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host cloud-26 11:42:52,983 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[hidden email]:51449] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 11:44:32,976 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more . 11:44:32,984 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: Requesting the next InputSplit failed. at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) ... 4 more . On 28 Jan 2015, at 12:08, Robert Metzger <[hidden email]> wrote: > Hi, > > it seems that you are not subscribed to our mailing list, so I had to > manually accept your mail. Would be good if you could subscribe. > > Can you send us also the log output of the JobManager? > If your YARN cluster has log aggregation activated, you can retrieve the > logs of a stopped YARN session using: > yarn logs -applicationId <AppId> > > watch out for the jobmanager-main.log or so file. > > I suspect that there has been an exception on the JobManager. > > Best, > Robert > > > > On Wed, Jan 28, 2015 at 12:01 PM, Bruecke, Christoph < > [hidden email]> wrote: > >> Hi, >> >> I have written a job that reads a SequenceFile from HDFS using the >> Hadoop-Compatibility add-on. Doing so results in a TimeoutException. I’m >> using flink-0.9-SNAPSHOT with PR 342 ( >> https://github.com/apache/flink/pull/342 ). Furthermore I’m running flink >> on yarn with two TM using flink-yarn-0.9-SNAPSHOT/bin/yarn-session.sh -n 2. >> >> Is this a bug or is there something wrong with the configuration? >> >> 01/28/2015 11:42:52 Job execution switched to status RUNNING. >> 01/28/2015 11:42:52 CHAIN DataSource (at >> createInput(ExecutionEnvironment.java:426) >> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to SCHEDULED >> 01/28/2015 11:42:52 CHAIN DataSource (at >> createInput(ExecutionEnvironment.java:426) >> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to DEPLOYING >> 01/28/2015 11:42:52 CHAIN DataSource (at >> createInput(ExecutionEnvironment.java:426) >> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to RUNNING >> 01/28/2015 11:44:32 CHAIN DataSource (at >> createInput(ExecutionEnvironment.java:426) >> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to FAILED >> java.lang.RuntimeException: Requesting the next InputSplit failed. >> at >> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) >> at >> org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) >> at >> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) >> at >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.util.concurrent.TimeoutException: Futures timed out after >> [100 seconds] >> at >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> at >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> at >> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> at >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> at scala.concurrent.Await$.result(package.scala:107) >> at >> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) >> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) >> at >> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) >> ... 4 more >> >> 01/28/2015 11:44:32 Job execution switched to status FAILING. >> 01/28/2015 11:44:32 GroupReduce (GroupReduce at >> main(ThiaziParser.java:40))(1/1) switched to CANCELED >> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// >> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/authors) >> - UTF-8)(1/1) switched to CANCELED >> 01/28/2015 11:44:32 CHAIN GroupReduce (GroupReduce at >> main(ThiaziParser.java:74)) -> Filter (Filter at >> main(ThiaziParser.java:97))(1/1) switched to CANCELED >> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// >> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/posts) - >> UTF-8)(1/1) switched to CANCELED >> 01/28/2015 11:44:32 CHAIN FlatMap (FlatMap at >> main(ThiaziParser.java:126)) -> Combine(SUM(1), at >> main(ThiaziParser.java:140)(1/1) switched to CANCELED >> 01/28/2015 11:44:32 Reduce (SUM(1), at >> main(ThiaziParser.java:140)(1/1) switched to CANCELED >> 01/28/2015 11:44:32 DataSink(CsvOutputFormat (path: hdfs:// >> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/wordcount, >> delimiter: ,))(1/1) switched to CANCELED >> 01/28/2015 11:44:32 GroupReduce (GroupReduce at >> main(ThiaziParser.java:106))(1/1) switched to CANCELED >> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// >> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/threads) >> - UTF-8)(1/1) switched to CANCELED >> 01/28/2015 11:44:32 Job execution switched to status FAILED. >> Error: The program execution failed: java.lang.RuntimeException: >> Requesting the next InputSplit failed. >> at >> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) >> at >> org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) >> at >> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) >> at >> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.util.concurrent.TimeoutException: Futures timed out after >> [100 seconds] >> at >> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >> at >> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >> at >> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >> at >> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >> at scala.concurrent.Await$.result(package.scala:107) >> at >> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) >> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) >> at >> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) >> ... 4 more >> >> >> |
I might add that the error only occurs when running with the RemoteExecutor regardless of the number of TM. Starting the job in IntelliJ with the LocalExecutor with dop 1 works just fine.
Best, Christoph On 28 Jan 2015, at 12:17, Bruecke, Christoph <[hidden email]> wrote: > Hi Robert, > > thanks for the quick response. Here is the jobmanager-main.log: > > PS: I’m subscribed now. > > 11:09:16,144 INFO org.apache.flink.yarn.ApplicationMaster$ - YARN daemon runs as hadoop setting user to execute Flink ApplicationMaster/JobManager to hadoop > 11:09:16,199 INFO org.apache.flink.yarn.Utils - Found YARN_CONF_DIR, adding it to configuration > 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ - Start job manager for yarn > 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ - Config path: /data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001. > 11:09:17,179 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started > 11:09:17,271 INFO Remoting - Starting remoting > 11:09:17,556 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://[hidden email]:42643] > 11:09:17,569 INFO org.apache.flink.yarn.ApplicationMaster$ - Start job manager actor. > 11:09:17,570 INFO org.apache.flink.yarn.ApplicationMaster$ - Generate configuration file for application master. > 11:09:17,579 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Starting job manager at akka://flink/user/jobmanager. > 11:09:17,590 INFO org.apache.flink.yarn.ApplicationMaster$ - Start yarn session on job manager. > 11:09:17,591 INFO org.apache.flink.yarn.ApplicationMaster$ - Application Master properly initiated. Await termination of actor system. > 11:09:17,601 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server on port 35605 > 11:09:17,616 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Started job manager. Waiting for incoming messages. > 11:09:17,621 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer - Setting up web info server, using web-root directoryjar:file:/data/4/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/filecache/11/flink-dist-0.9-SNAPSHOT-yarn-uberjar.jar!/web-docs-infoserver. > 11:09:17,621 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer - Web info server will display information about flink job-manager on localhost, port 8081. > 11:09:17,733 INFO org.apache.flink.runtime.jobmanager.web.WebInfoServer - Starting web info server for JobManager on port 8081 > 11:09:17,734 INFO org.eclipse.jetty.util.log - jetty-0.9-SNAPSHOT > 11:09:17,770 INFO org.eclipse.jetty.util.log - Started SelectChannelConnector@0.0.0.0:8081 > 11:09:17,784 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Start yarn session. > 11:09:17,784 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Requesting 2 task managers. > 11:09:18,414 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at cloud-11/130.149.21.15:8030 > 11:09:18,423 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-nodemanagers-proxies : 500 > 11:09:18,425 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Registering ApplicationMaster with tracking url http://cloud-29.dima.tu-berlin.de:8081. > 11:09:18,787 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Requesting TaskManager container 0. > 11:09:18,794 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Requesting TaskManager container 1. > 11:09:18,903 INFO org.apache.flink.yarn.Utils - Copying from file:/data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001/flink-conf-modified.yaml to hdfs://cloud-11.dima.tu-berlin.de:60010/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml > 11:09:19,280 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "cloud-11.dima.tu-berlin.de" port: 60010 file: "/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml" } size: 3121 timestamp: 1422439759181 type: FILE visibility: APPLICATION > 11:09:19,286 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Create container launch context. > 11:09:19,300 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Starting TM with command=$JAVA_HOME/bin/java -Xmx819m -Dlog.file="<LOG_DIR>/taskmanager.log" -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.appMaster.YarnTaskManagerRunner --configDir . 1> <LOG_DIR>/taskmanager-stdout.log 2> <LOG_DIR>/taskmanager-stderr.log > 11:09:19,619 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Got new container for TM container_1420727594991_0068_01_000002 on host cloud-26.dima.tu-berlin.de > 11:09:19,621 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Launching container #1. > 11:09:19,622 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : cloud-26.dima.tu-berlin.de:8045 > 11:09:19,828 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Got new container for TM container_1420727594991_0068_01_000003 on host cloud-31.dima.tu-berlin.de > 11:09:19,829 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Launching container #2. > 11:09:19,831 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : cloud-31.dima.tu-berlin.de:8045 > 11:09:25,748 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at akka.tcp://[hidden email]:51449/user/taskmanager as ca731dbbfe941a63077c9c148254607c. Current number of registered hosts is 1. > 11:09:25,845 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at akka.tcp://[hidden email]:43200/user/taskmanager as 7bf3aa638e00682239c9d8460d142191. Current number of registered hosts is 2. > 11:11:13,837 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://CliFrontendActorSystem@130.149.21.15:50292] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > 11:14:30,892 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Received job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015). > 11:14:30,997 INFO org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.interval is deprecated. Instead, use mapreduce.job.end-notification.retry.interval > 11:14:30,998 INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > 11:14:30,999 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use mapreduce.jobtracker.retiredjobs.cache.size > 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.reduces is deprecated. Instead, use mapreduce.task.profile.reduces > 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use mapreduce.job.jvm.numtasks > 11:14:31,001 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative > 11:14:31,003 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.http.address is deprecated. Instead, use mapreduce.tasktracker.http.address > 11:14:31,004 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir > 11:14:31,005 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.report.address is deprecated. Instead, use mapreduce.tasktracker.report.address > 11:14:31,006 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.healthChecker.interval is deprecated. Instead, use mapreduce.tasktracker.healthchecker.interval > 11:14:31,007 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.child.tmp is deprecated. Instead, use mapreduce.task.tmp.dir > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.taskmemorymanager.monitoring-interval is deprecated. Instead, use mapreduce.tasktracker.taskmemorymanager.monitoringinterval > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.shuffle.connect.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.connect.timeout > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.speculativeCap is deprecated. Instead, use mapreduce.job.speculative.speculativecap > 11:14:31,009 INFO org.apache.hadoop.conf.Configuration.deprecation - io.sort.spill.percent is deprecated. Instead, use mapreduce.map.sort.spill.percent > 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.shuffle.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.shuffle.input.buffer.percent > 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.map.max.skip.records is deprecated. Instead, use mapreduce.map.skip.maxrecords > 11:14:31,011 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile.maps is deprecated. Instead, use mapreduce.task.profile.maps > 11:14:31,012 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.merge.recordsBeforeProgress is deprecated. Instead, use mapreduce.task.merge.progress.records > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.retain.hours is deprecated. Instead, use mapreduce.job.userlog.retain.hours > 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress > 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use mapreduce.job.speculative.slownodethreshold > 11:14:31,015 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, use mapreduce.tasktracker.reduce.tasks.maximum > 11:14:31,017 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level > 11:14:31,018 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.restart.recover is deprecated. Instead, use mapreduce.jobtracker.restart.recover > 11:14:31,020 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.inmem.merge.threshold is deprecated. Instead, use mapreduce.reduce.merge.inmem.threshold > 11:14:31,021 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces > 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.acls.enabled is deprecated. Instead, use mapreduce.cluster.acls.enabled > 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.dns.nameserver is deprecated. Instead, use mapreduce.tasktracker.dns.nameserver > 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication > 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative > 11:14:31,025 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.shuffle.merge.percent is deprecated. Instead, use mapreduce.reduce.shuffle.merge.percent > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.lru.cache.size > 11:14:31,029 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename > 11:14:31,030 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.hours > 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout > 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use mapreduce.tasktracker.map.tasks.maximum > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation - jobclient.completion.poll.interval is deprecated. Instead, use mapreduce.client.completion.pollinterval > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.dir > 11:14:31,033 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.slowstart.completed.maps is deprecated. Instead, use mapreduce.job.reduce.slowstart.completedmaps > 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb > 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation - dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode > 11:14:31,037 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.instrumentation is deprecated. Instead, use mapreduce.jobtracker.instrumentation > 11:14:31,038 INFO org.apache.hadoop.conf.Configuration.deprecation - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl > 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type > 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.attempts.to.start.skipping is deprecated. Instead, use mapreduce.task.skip.start.attempts > 11:14:31,040 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.tracker.task-controller is deprecated. Instead, use mapreduce.tasktracker.taskcontroller > 11:14:31,041 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.userlog.limit.kb is deprecated. Instead, use mapreduce.task.userlog.limit.kb > 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed > 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps > 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir.minspacekill is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacekill > 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.jobtracker.split.metainfo.maxsize is deprecated. Instead, use mapreduce.job.split.metainfo.maxsize > 11:14:31,045 INFO org.apache.hadoop.conf.Configuration.deprecation - jobclient.progress.monitor.poll.interval is deprecated. Instead, use mapreduce.client.progressmonitor.pollinterval > 11:14:31,046 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize > 11:14:31,047 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec > 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.profile is deprecated. Instead, use mapreduce.task.profile > 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.parallel.copies is deprecated. Instead, use mapreduce.reduce.shuffle.parallelcopies > 11:14:31,049 INFO org.apache.hadoop.conf.Configuration.deprecation - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor > 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir > 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout > 11:14:31,052 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.heartbeats.in.second is deprecated. Instead, use mapreduce.jobtracker.heartbeats.in.second > 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.cache.levels is deprecated. Instead, use mapreduce.jobtracker.taskcache.levels > 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.map.output.compression.codec is deprecated. Instead, use mapreduce.map.output.compress.codec > 11:14:31,054 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.dns.interface is deprecated. Instead, use mapreduce.tasktracker.dns.interface > 11:14:31,055 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.max.tracker.failures is deprecated. Instead, use mapreduce.job.maxtaskfailures.per.tracker > 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation - dfs.df.interval is deprecated. Instead, use fs.df.interval > 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead, use mapreduce.tasktracker.tasks.sleeptimebeforesigkill > 11:14:31,057 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.max.tracker.blacklists is deprecated. Instead, use mapreduce.jobtracker.tasktracker.maxblacklists > 11:14:31,058 INFO org.apache.hadoop.conf.Configuration.deprecation - jobclient.output.filter is deprecated. Instead, use mapreduce.client.output.filter > 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation - job.end.retry.attempts is deprecated. Instead, use mapreduce.job.end-notification.retry.attempts > 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.taskScheduler is deprecated. Instead, use mapreduce.jobtracker.taskscheduler > 11:14:31,060 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.speculative.execution.slowTaskThreshold is deprecated. Instead, use mapreduce.job.speculative.slowtaskthreshold > 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.indexcache.mb is deprecated. Instead, use mapreduce.tasktracker.indexcache.mb > 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation - tasktracker.http.threads is deprecated. Instead, use mapreduce.tasktracker.http.threads > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.handler.count is deprecated. Instead, use mapreduce.jobtracker.handler.count > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation - keep.failed.task.files is deprecated. Instead, use mapreduce.task.files.preserve.failedtasks > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress > 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.job.history.block.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.block.size > 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.skip.reduce.max.skip.groups is deprecated. Instead, use mapreduce.reduce.skip.maxgroups > 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args > 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS > 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.local.dir.minspacestart is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacestart > 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use mapreduce.jobtracker.maxtasks.perjob > 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.max.attempts is deprecated. Instead, use mapreduce.reduce.maxattempts > 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address > 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.shuffle.read.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.read.timeout > 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.instrumentation is deprecated. Instead, use mapreduce.tasktracker.instrumentation > 11:14:31,070 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.tasktracker.expiry.interval is deprecated. Instead, use mapreduce.jobtracker.expire.trackers.interval > 11:14:31,071 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.active > 11:14:31,074 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir > 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation - hadoop.native.lib is deprecated. Instead, use io.native.lib.available > 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max > 11:14:31,076 INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.input.buffer.percent > 11:14:31,149 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 > 11:14:31,177 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Scheduling job Flink Java Job at Wed Jan 28 11:14:28 CET 2015. > 11:14:31,188 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to cloud-26 > 11:14:31,202 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015) changed to RUNNING. > 11:14:32,525 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host cloud-26 > 11:14:32,603 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[hidden email]:51449] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > 11:16:12,574 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015) changed to FAILINGjava.lang.RuntimeException: Requesting the next InputSplit failed. > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > . > 11:16:12,584 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan 28 11:14:28 CET 2015) changed to FAILEDjava.lang.RuntimeException: Requesting the next InputSplit failed. > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > . > 11:17:53,055 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Received job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015). > 11:17:53,258 INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > 11:17:53,312 INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS > 11:17:53,320 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max > 11:17:53,347 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 > 11:17:53,352 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to cloud-31 > 11:17:53,353 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Scheduling job Flink Java Job at Wed Jan 28 11:17:50 CET 2015. > 11:17:53,354 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015) changed to RUNNING. > 11:17:54,646 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host cloud-31 > 11:17:54,715 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[hidden email]:43200] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > 11:19:34,684 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: Requesting the next InputSplit failed. > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > . > 11:19:34,697 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan 28 11:17:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: Requesting the next InputSplit failed. > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > . > 11:42:52,441 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Received job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015). > 11:42:52,650 INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > 11:42:52,705 INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS > 11:42:52,713 INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max > 11:42:52,740 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 > 11:42:52,747 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Scheduling job Flink Java Job at Wed Jan 28 11:42:50 CET 2015. > 11:42:52,747 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to cloud-26 > 11:42:52,750 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015) changed to RUNNING. > 11:42:52,956 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host cloud-26 > 11:42:52,983 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[hidden email]:51449] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > 11:44:32,976 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: Requesting the next InputSplit failed. > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > . > 11:44:32,984 INFO org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan 28 11:42:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: Requesting the next InputSplit failed. > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > at org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after [100 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > at org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > ... 4 more > . > > On 28 Jan 2015, at 12:08, Robert Metzger <[hidden email]> wrote: > >> Hi, >> >> it seems that you are not subscribed to our mailing list, so I had to >> manually accept your mail. Would be good if you could subscribe. >> >> Can you send us also the log output of the JobManager? >> If your YARN cluster has log aggregation activated, you can retrieve the >> logs of a stopped YARN session using: >> yarn logs -applicationId <AppId> >> >> watch out for the jobmanager-main.log or so file. >> >> I suspect that there has been an exception on the JobManager. >> >> Best, >> Robert >> >> >> >> On Wed, Jan 28, 2015 at 12:01 PM, Bruecke, Christoph < >> [hidden email]> wrote: >> >>> Hi, >>> >>> I have written a job that reads a SequenceFile from HDFS using the >>> Hadoop-Compatibility add-on. Doing so results in a TimeoutException. I’m >>> using flink-0.9-SNAPSHOT with PR 342 ( >>> https://github.com/apache/flink/pull/342 ). Furthermore I’m running flink >>> on yarn with two TM using flink-yarn-0.9-SNAPSHOT/bin/yarn-session.sh -n 2. >>> >>> Is this a bug or is there something wrong with the configuration? >>> >>> 01/28/2015 11:42:52 Job execution switched to status RUNNING. >>> 01/28/2015 11:42:52 CHAIN DataSource (at >>> createInput(ExecutionEnvironment.java:426) >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to SCHEDULED >>> 01/28/2015 11:42:52 CHAIN DataSource (at >>> createInput(ExecutionEnvironment.java:426) >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to DEPLOYING >>> 01/28/2015 11:42:52 CHAIN DataSource (at >>> createInput(ExecutionEnvironment.java:426) >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to RUNNING >>> 01/28/2015 11:44:32 CHAIN DataSource (at >>> createInput(ExecutionEnvironment.java:426) >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to FAILED >>> java.lang.RuntimeException: Requesting the next InputSplit failed. >>> at >>> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) >>> at >>> org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) >>> at >>> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) >>> at >>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out after >>> [100 seconds] >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >>> at >>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >>> at >>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >>> at scala.concurrent.Await$.result(package.scala:107) >>> at >>> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) >>> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) >>> at >>> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) >>> ... 4 more >>> >>> 01/28/2015 11:44:32 Job execution switched to status FAILING. >>> 01/28/2015 11:44:32 GroupReduce (GroupReduce at >>> main(ThiaziParser.java:40))(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// >>> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/authors) >>> - UTF-8)(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 CHAIN GroupReduce (GroupReduce at >>> main(ThiaziParser.java:74)) -> Filter (Filter at >>> main(ThiaziParser.java:97))(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// >>> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/posts) - >>> UTF-8)(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 CHAIN FlatMap (FlatMap at >>> main(ThiaziParser.java:126)) -> Combine(SUM(1), at >>> main(ThiaziParser.java:140)(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 Reduce (SUM(1), at >>> main(ThiaziParser.java:140)(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 DataSink(CsvOutputFormat (path: hdfs:// >>> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/wordcount, >>> delimiter: ,))(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 GroupReduce (GroupReduce at >>> main(ThiaziParser.java:106))(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// >>> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/threads) >>> - UTF-8)(1/1) switched to CANCELED >>> 01/28/2015 11:44:32 Job execution switched to status FAILED. >>> Error: The program execution failed: java.lang.RuntimeException: >>> Requesting the next InputSplit failed. >>> at >>> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) >>> at >>> org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) >>> at >>> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) >>> at >>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out after >>> [100 seconds] >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >>> at >>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >>> at >>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >>> at scala.concurrent.Await$.result(package.scala:107) >>> at >>> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) >>> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) >>> at >>> org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) >>> ... 4 more >>> >>> >>> > |
I see the following line:
11:14:32,603 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp:// [hidden email]:51449] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. Does that mean that the machines have lost connection? @Till What is your take on this? On Wed, Jan 28, 2015 at 7:07 AM, Bruecke, Christoph < [hidden email]> wrote: > I might add that the error only occurs when running with the > RemoteExecutor regardless of the number of TM. Starting the job in IntelliJ > with the LocalExecutor with dop 1 works just fine. > > Best, > Christoph > > On 28 Jan 2015, at 12:17, Bruecke, Christoph < > [hidden email]> wrote: > > > Hi Robert, > > > > thanks for the quick response. Here is the jobmanager-main.log: > > > > PS: I’m subscribed now. > > > > 11:09:16,144 INFO org.apache.flink.yarn.ApplicationMaster$ > - YARN daemon runs as hadoop setting user to execute Flink > ApplicationMaster/JobManager to hadoop > > 11:09:16,199 INFO org.apache.flink.yarn.Utils > - Found YARN_CONF_DIR, adding it to configuration > > 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ > - Start job manager for yarn > > 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ > - Config path: > /data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001. > > 11:09:17,179 INFO akka.event.slf4j.Slf4jLogger > - Slf4jLogger started > > 11:09:17,271 INFO Remoting > - Starting remoting > > 11:09:17,556 INFO Remoting > - Remoting started; listening on addresses :[akka.tcp:// > [hidden email]:42643] > > 11:09:17,569 INFO org.apache.flink.yarn.ApplicationMaster$ > - Start job manager actor. > > 11:09:17,570 INFO org.apache.flink.yarn.ApplicationMaster$ > - Generate configuration file for application master. > > 11:09:17,579 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Starting job manager at akka://flink/user/jobmanager. > > 11:09:17,590 INFO org.apache.flink.yarn.ApplicationMaster$ > - Start yarn session on job manager. > > 11:09:17,591 INFO org.apache.flink.yarn.ApplicationMaster$ > - Application Master properly initiated. Await termination of actor > system. > > 11:09:17,601 INFO org.apache.flink.runtime.blob.BlobServer > - Started BLOB server on port 35605 > > 11:09:17,616 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Started job manager. Waiting for incoming messages. > > 11:09:17,621 INFO > org.apache.flink.runtime.jobmanager.web.WebInfoServer - Setting up > web info server, using web-root > directoryjar:file:/data/4/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/filecache/11/flink-dist-0.9-SNAPSHOT-yarn-uberjar.jar!/web-docs-infoserver. > > 11:09:17,621 INFO > org.apache.flink.runtime.jobmanager.web.WebInfoServer - Web info > server will display information about flink job-manager on localhost, port > 8081. > > 11:09:17,733 INFO > org.apache.flink.runtime.jobmanager.web.WebInfoServer - Starting > web info server for JobManager on port 8081 > > 11:09:17,734 INFO org.eclipse.jetty.util.log > - jetty-0.9-SNAPSHOT > > 11:09:17,770 INFO org.eclipse.jetty.util.log > - Started SelectChannelConnector@0.0.0.0:8081 > > 11:09:17,784 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Start yarn session. > > 11:09:17,784 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Requesting 2 task managers. > > 11:09:18,414 INFO org.apache.hadoop.yarn.client.RMProxy > - Connecting to ResourceManager at cloud-11/130.149.21.15:8030 > > 11:09:18,423 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > yarn.client.max-nodemanagers-proxies : 500 > > 11:09:18,425 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Registering ApplicationMaster with tracking url > http://cloud-29.dima.tu-berlin.de:8081. > > 11:09:18,787 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Requesting TaskManager container 0. > > 11:09:18,794 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Requesting TaskManager container 1. > > 11:09:18,903 INFO org.apache.flink.yarn.Utils > - Copying from > file:/data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001/flink-conf-modified.yaml > to hdfs:// > cloud-11.dima.tu-berlin.de:60010/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml > > 11:09:19,280 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Prepared local resource for modified yaml: resource { scheme: "hdfs" > host: "cloud-11.dima.tu-berlin.de" port: 60010 file: > "/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml" > } size: 3121 timestamp: 1422439759181 type: FILE visibility: APPLICATION > > 11:09:19,286 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Create container launch context. > > 11:09:19,300 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Starting TM with command=$JAVA_HOME/bin/java -Xmx819m > -Dlog.file="<LOG_DIR>/taskmanager.log" > -Dlogback.configurationFile=file:logback.xml > -Dlog4j.configuration=file:log4j.properties > org.apache.flink.yarn.appMaster.YarnTaskManagerRunner --configDir . 1> > <LOG_DIR>/taskmanager-stdout.log 2> <LOG_DIR>/taskmanager-stderr.log > > 11:09:19,619 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Got new container for TM container_1420727594991_0068_01_000002 on host > cloud-26.dima.tu-berlin.de > > 11:09:19,621 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Launching container #1. > > 11:09:19,622 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : cloud-26.dima.tu-berlin.de:8045 > > 11:09:19,828 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Got new container for TM container_1420727594991_0068_01_000003 on host > cloud-31.dima.tu-berlin.de > > 11:09:19,829 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Launching container #2. > > 11:09:19,831 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : cloud-31.dima.tu-berlin.de:8045 > > 11:09:25,748 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at akka.tcp:// > [hidden email]:51449/user/taskmanager as > ca731dbbfe941a63077c9c148254607c. Current number of registered hosts is 1. > > 11:09:25,845 INFO org.apache.flink.runtime.instance.InstanceManager > - Registered TaskManager at akka.tcp:// > [hidden email]:43200/user/taskmanager as > 7bf3aa638e00682239c9d8460d142191. Current number of registered hosts is 2. > > 11:11:13,837 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > CliFrontendActorSystem@130.149.21.15:50292] has failed, address is now > gated for [5000] ms. Reason is: [Disassociated]. > > 11:14:30,892 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Received job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan > 28 11:14:28 CET 2015). > > 11:14:30,997 INFO org.apache.hadoop.conf.Configuration.deprecation > - job.end.retry.interval is deprecated. Instead, use > mapreduce.job.end-notification.retry.interval > > 11:14:30,998 INFO org.apache.hadoop.conf.Configuration.deprecation > - io.bytes.per.checksum is deprecated. Instead, use > dfs.bytes-per-checksum > > 11:14:30,999 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, > use mapreduce.jobtracker.retiredjobs.cache.size > > 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.profile.reduces is deprecated. Instead, use > mapreduce.task.profile.reduces > > 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use > mapreduce.job.jvm.numtasks > > 11:14:31,001 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.reduce.tasks.speculative.execution is deprecated. Instead, > use mapreduce.reduce.speculative > > 11:14:31,003 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.tracker.http.address is deprecated. Instead, use > mapreduce.tasktracker.http.address > > 11:14:31,004 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.system.dir is deprecated. Instead, use > mapreduce.jobtracker.system.dir > > 11:14:31,005 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.tracker.report.address is deprecated. Instead, use > mapreduce.tasktracker.report.address > > 11:14:31,006 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.healthChecker.interval is deprecated. Instead, use > mapreduce.tasktracker.healthchecker.interval > > 11:14:31,007 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.child.tmp is deprecated. Instead, use > mapreduce.task.tmp.dir > > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.taskmemorymanager.monitoring-interval is > deprecated. Instead, use > mapreduce.tasktracker.taskmemorymanager.monitoringinterval > > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.shuffle.connect.timeout is deprecated. Instead, use > mapreduce.reduce.shuffle.connect.timeout > > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.speculative.execution.speculativeCap is deprecated. > Instead, use mapreduce.job.speculative.speculativecap > > 11:14:31,009 INFO org.apache.hadoop.conf.Configuration.deprecation > - io.sort.spill.percent is deprecated. Instead, use > mapreduce.map.sort.spill.percent > > 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.shuffle.input.buffer.percent is deprecated. Instead, > use mapreduce.reduce.shuffle.input.buffer.percent > > 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.skip.map.max.skip.records is deprecated. Instead, use > mapreduce.map.skip.maxrecords > > 11:14:31,011 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.profile.maps is deprecated. Instead, use > mapreduce.task.profile.maps > > 11:14:31,012 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.local.dir is deprecated. Instead, use > mapreduce.cluster.local.dir > > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.merge.recordsBeforeProgress is deprecated. Instead, use > mapreduce.task.merge.progress.records > > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.http.address is deprecated. Instead, use > mapreduce.jobtracker.http.address > > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.userlog.retain.hours is deprecated. Instead, use > mapreduce.job.userlog.retain.hours > > 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.compress.map.output is deprecated. Instead, use > mapreduce.map.output.compress > > 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.speculative.execution.slowNodeThreshold is deprecated. > Instead, use mapreduce.job.speculative.slownodethreshold > > 11:14:31,015 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, > use mapreduce.tasktracker.reduce.tasks.maximum > > 11:14:31,017 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.reduce.child.log.level is deprecated. Instead, use > mapreduce.reduce.log.level > > 11:14:31,018 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.jobtracker.restart.recover is deprecated. Instead, use > mapreduce.jobtracker.restart.recover > > 11:14:31,020 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.inmem.merge.threshold is deprecated. Instead, use > mapreduce.reduce.merge.inmem.threshold > > 11:14:31,021 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.reduce.tasks is deprecated. Instead, use > mapreduce.job.reduces > > 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.acls.enabled is deprecated. Instead, use > mapreduce.cluster.acls.enabled > > 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.dns.nameserver is deprecated. Instead, use > mapreduce.tasktracker.dns.nameserver > > 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.submit.replication is deprecated. Instead, use > mapreduce.client.submit.file.replication > > 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.map.tasks.speculative.execution is deprecated. Instead, > use mapreduce.map.speculative > > 11:14:31,025 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.map.child.log.level is deprecated. Instead, use > mapreduce.map.log.level > > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.map.max.attempts is deprecated. Instead, use > mapreduce.map.maxattempts > > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.shuffle.merge.percent is deprecated. Instead, use > mapreduce.reduce.shuffle.merge.percent > > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.jobhistory.lru.cache.size is deprecated. > Instead, use mapreduce.jobtracker.jobhistory.lru.cache.size > > 11:14:31,029 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.queue.name is deprecated. Instead, use > mapreduce.job.queuename > > 11:14:31,030 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.persist.jobstatus.hours is deprecated. > Instead, use mapreduce.jobtracker.persist.jobstatus.hours > > 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.healthChecker.script.timeout is deprecated. Instead, use > mapreduce.tasktracker.healthchecker.script.timeout > > 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use > mapreduce.tasktracker.map.tasks.maximum > > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.reduce.markreset.buffer.percent is deprecated. > Instead, use mapreduce.reduce.markreset.buffer.percent > > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation > - jobclient.completion.poll.interval is deprecated. Instead, use > mapreduce.client.completion.pollinterval > > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, > use mapreduce.jobtracker.persist.jobstatus.dir > > 11:14:31,033 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.reduce.slowstart.completed.maps is deprecated. Instead, > use mapreduce.job.reduce.slowstart.completedmaps > > 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation > - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb > > 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation > - dfs.umaskmode is deprecated. Instead, use > fs.permissions.umask-mode > > 11:14:31,037 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.jobtracker.instrumentation is deprecated. Instead, use > mapreduce.jobtracker.instrumentation > > 11:14:31,038 INFO org.apache.hadoop.conf.Configuration.deprecation > - topology.node.switch.mapping.impl is deprecated. Instead, use > net.topology.node.switch.mapping.impl > > 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.output.compression.type is deprecated. Instead, use > mapreduce.output.fileoutputformat.compress.type > > 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.skip.attempts.to.start.skipping is deprecated. Instead, > use mapreduce.task.skip.start.attempts > > 11:14:31,040 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.tracker.task-controller is deprecated. Instead, use > mapreduce.tasktracker.taskcontroller > > 11:14:31,041 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.userlog.limit.kb is deprecated. Instead, use > mapreduce.task.userlog.limit.kb > > 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.committer.job.setup.cleanup.needed is deprecated. Instead, > use mapreduce.job.committer.setup.cleanup.needed > > 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps > > 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.local.dir.minspacekill is deprecated. Instead, use > mapreduce.tasktracker.local.dir.minspacekill > > 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapreduce.jobtracker.split.metainfo.maxsize is deprecated. > Instead, use mapreduce.job.split.metainfo.maxsize > > 11:14:31,045 INFO org.apache.hadoop.conf.Configuration.deprecation > - jobclient.progress.monitor.poll.interval is deprecated. Instead, > use mapreduce.client.progressmonitor.pollinterval > > 11:14:31,046 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.min.split.size is deprecated. Instead, use > mapreduce.input.fileinputformat.split.minsize > > 11:14:31,047 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.output.compression.codec is deprecated. Instead, use > mapreduce.output.fileoutputformat.compress.codec > > 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.profile is deprecated. Instead, use > mapreduce.task.profile > > 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.reduce.parallel.copies is deprecated. Instead, use > mapreduce.reduce.shuffle.parallelcopies > > 11:14:31,049 INFO org.apache.hadoop.conf.Configuration.deprecation > - io.sort.factor is deprecated. Instead, use > mapreduce.task.io.sort.factor > > 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.input.dir is deprecated. Instead, use > mapreduce.input.fileinputformat.inputdir > > 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.timeout is deprecated. Instead, use > mapreduce.task.timeout > > 11:14:31,052 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.heartbeats.in.second is deprecated. Instead, use > mapreduce.jobtracker.heartbeats.in.second > > 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.task.cache.levels is deprecated. Instead, use > mapreduce.jobtracker.taskcache.levels > > 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.map.output.compression.codec is deprecated. Instead, use > mapreduce.map.output.compress.codec > > 11:14:31,054 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.dns.interface is deprecated. Instead, use > mapreduce.tasktracker.dns.interface > > 11:14:31,055 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.max.tracker.failures is deprecated. Instead, use > mapreduce.job.maxtaskfailures.per.tracker > > 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation > - dfs.df.interval is deprecated. Instead, use fs.df.interval > > 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. > Instead, use mapreduce.tasktracker.tasks.sleeptimebeforesigkill > > 11:14:31,057 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.max.tracker.blacklists is deprecated. Instead, use > mapreduce.jobtracker.tasktracker.maxblacklists > > 11:14:31,058 INFO org.apache.hadoop.conf.Configuration.deprecation > - jobclient.output.filter is deprecated. Instead, use > mapreduce.client.output.filter > > 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation > - job.end.retry.attempts is deprecated. Instead, use > mapreduce.job.end-notification.retry.attempts > > 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.jobtracker.taskScheduler is deprecated. Instead, use > mapreduce.jobtracker.taskscheduler > > 11:14:31,060 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.speculative.execution.slowTaskThreshold is deprecated. > Instead, use mapreduce.job.speculative.slowtaskthreshold > > 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.indexcache.mb is deprecated. Instead, use > mapreduce.tasktracker.indexcache.mb > > 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation > - tasktracker.http.threads is deprecated. Instead, use > mapreduce.tasktracker.http.threads > > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.handler.count is deprecated. Instead, use > mapreduce.jobtracker.handler.count > > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation > - keep.failed.task.files is deprecated. Instead, use > mapreduce.task.files.preserve.failedtasks > > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.output.compress is deprecated. Instead, use > mapreduce.output.fileoutputformat.compress > > 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.jobtracker.job.history.block.size is deprecated. Instead, > use mapreduce.jobtracker.jobhistory.block.size > > 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.skip.reduce.max.skip.groups is deprecated. Instead, use > mapreduce.reduce.skip.maxgroups > > 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation > - topology.script.number.args is deprecated. Instead, use > net.topology.script.number.args > > 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation > - fs.default.name is deprecated. Instead, use fs.defaultFS > > 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.local.dir.minspacestart is deprecated. Instead, use > mapreduce.tasktracker.local.dir.minspacestart > > 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use > mapreduce.jobtracker.maxtasks.perjob > > 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.reduce.max.attempts is deprecated. Instead, use > mapreduce.reduce.maxattempts > > 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker is deprecated. Instead, use > mapreduce.jobtracker.address > > 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.shuffle.read.timeout is deprecated. Instead, use > mapreduce.reduce.shuffle.read.timeout > > 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.instrumentation is deprecated. Instead, use > mapreduce.tasktracker.instrumentation > > 11:14:31,070 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.tasktracker.expiry.interval is deprecated. Instead, use > mapreduce.jobtracker.expire.trackers.interval > > 11:14:31,071 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.tracker.persist.jobstatus.active is deprecated. > Instead, use mapreduce.jobtracker.persist.jobstatus.active > > 11:14:31,074 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.temp.dir is deprecated. Instead, use > mapreduce.cluster.temp.dir > > 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation > - hadoop.native.lib is deprecated. Instead, use > io.native.lib.available > > 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapreduce.job.counters.limit is deprecated. Instead, use > mapreduce.job.counters.max > > 11:14:31,076 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapred.job.reduce.input.buffer.percent is deprecated. Instead, > use mapreduce.reduce.input.buffer.percent > > 11:14:31,149 INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input > paths to process : 1 > > 11:14:31,177 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Scheduling job Flink Java Job at Wed Jan 28 11:14:28 CET 2015. > > 11:14:31,188 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to > cloud-26 > > 11:14:31,202 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan > 28 11:14:28 CET 2015) changed to RUNNING. > > 11:14:32,525 INFO > org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning > remote split to host cloud-26 > > 11:14:32,603 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > [hidden email]:51449] has failed, address is now gated > for [5000] ms. Reason is: [Disassociated]. > > 11:16:12,574 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan > 28 11:14:28 CET 2015) changed to FAILINGjava.lang.RuntimeException: > Requesting the next InputSplit failed. > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after [100 seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > ... 4 more > > . > > 11:16:12,584 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed Jan > 28 11:14:28 CET 2015) changed to FAILEDjava.lang.RuntimeException: > Requesting the next InputSplit failed. > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after [100 seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > ... 4 more > > . > > 11:17:53,055 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Received job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan > 28 11:17:50 CET 2015). > > 11:17:53,258 INFO org.apache.hadoop.conf.Configuration.deprecation > - io.bytes.per.checksum is deprecated. Instead, use > dfs.bytes-per-checksum > > 11:17:53,312 INFO org.apache.hadoop.conf.Configuration.deprecation > - fs.default.name is deprecated. Instead, use fs.defaultFS > > 11:17:53,320 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapreduce.job.counters.limit is deprecated. Instead, use > mapreduce.job.counters.max > > 11:17:53,347 INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input > paths to process : 1 > > 11:17:53,352 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to > cloud-31 > > 11:17:53,353 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Scheduling job Flink Java Job at Wed Jan 28 11:17:50 CET 2015. > > 11:17:53,354 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan > 28 11:17:50 CET 2015) changed to RUNNING. > > 11:17:54,646 INFO > org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning > remote split to host cloud-31 > > 11:17:54,715 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > [hidden email]:43200] has failed, address is now gated > for [5000] ms. Reason is: [Disassociated]. > > 11:19:34,684 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan > 28 11:17:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: > Requesting the next InputSplit failed. > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after [100 seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > ... 4 more > > . > > 11:19:34,697 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed Jan > 28 11:17:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: > Requesting the next InputSplit failed. > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after [100 seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > ... 4 more > > . > > 11:42:52,441 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Received job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan > 28 11:42:50 CET 2015). > > 11:42:52,650 INFO org.apache.hadoop.conf.Configuration.deprecation > - io.bytes.per.checksum is deprecated. Instead, use > dfs.bytes-per-checksum > > 11:42:52,705 INFO org.apache.hadoop.conf.Configuration.deprecation > - fs.default.name is deprecated. Instead, use fs.defaultFS > > 11:42:52,713 INFO org.apache.hadoop.conf.Configuration.deprecation > - mapreduce.job.counters.limit is deprecated. Instead, use > mapreduce.job.counters.max > > 11:42:52,740 INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input > paths to process : 1 > > 11:42:52,747 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Scheduling job Flink Java Job at Wed Jan 28 11:42:50 CET 2015. > > 11:42:52,747 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to > cloud-26 > > 11:42:52,750 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan > 28 11:42:50 CET 2015) changed to RUNNING. > > 11:42:52,956 INFO > org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning > remote split to host cloud-26 > > 11:42:52,983 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > [hidden email]:51449] has failed, address is now gated > for [5000] ms. Reason is: [Disassociated]. > > 11:44:32,976 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan > 28 11:42:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: > Requesting the next InputSplit failed. > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after [100 seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > ... 4 more > > . > > 11:44:32,984 INFO > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed Jan > 28 11:42:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: > Requesting the next InputSplit failed. > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > at > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > after [100 seconds] > > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > at > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > ... 4 more > > . > > > > On 28 Jan 2015, at 12:08, Robert Metzger <[hidden email]> wrote: > > > >> Hi, > >> > >> it seems that you are not subscribed to our mailing list, so I had to > >> manually accept your mail. Would be good if you could subscribe. > >> > >> Can you send us also the log output of the JobManager? > >> If your YARN cluster has log aggregation activated, you can retrieve the > >> logs of a stopped YARN session using: > >> yarn logs -applicationId <AppId> > >> > >> watch out for the jobmanager-main.log or so file. > >> > >> I suspect that there has been an exception on the JobManager. > >> > >> Best, > >> Robert > >> > >> > >> > >> On Wed, Jan 28, 2015 at 12:01 PM, Bruecke, Christoph < > >> [hidden email]> wrote: > >> > >>> Hi, > >>> > >>> I have written a job that reads a SequenceFile from HDFS using the > >>> Hadoop-Compatibility add-on. Doing so results in a TimeoutException. > I’m > >>> using flink-0.9-SNAPSHOT with PR 342 ( > >>> https://github.com/apache/flink/pull/342 ). Furthermore I’m running > flink > >>> on yarn with two TM using flink-yarn-0.9-SNAPSHOT/bin/yarn-session.sh > -n 2. > >>> > >>> Is this a bug or is there something wrong with the configuration? > >>> > >>> 01/28/2015 11:42:52 Job execution switched to status RUNNING. > >>> 01/28/2015 11:42:52 CHAIN DataSource (at > >>> createInput(ExecutionEnvironment.java:426) > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > SCHEDULED > >>> 01/28/2015 11:42:52 CHAIN DataSource (at > >>> createInput(ExecutionEnvironment.java:426) > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > DEPLOYING > >>> 01/28/2015 11:42:52 CHAIN DataSource (at > >>> createInput(ExecutionEnvironment.java:426) > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > RUNNING > >>> 01/28/2015 11:44:32 CHAIN DataSource (at > >>> createInput(ExecutionEnvironment.java:426) > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to FAILED > >>> java.lang.RuntimeException: Requesting the next InputSplit failed. > >>> at > >>> > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > >>> at > >>> > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > >>> at > >>> > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > >>> at > >>> > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > >>> at java.lang.Thread.run(Thread.java:745) > >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out > after > >>> [100 seconds] > >>> at > >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > >>> at > >>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > >>> at > >>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > >>> at > >>> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > >>> at scala.concurrent.Await$.result(package.scala:107) > >>> at > >>> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > >>> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > >>> at > >>> > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > >>> ... 4 more > >>> > >>> 01/28/2015 11:44:32 Job execution switched to status FAILING. > >>> 01/28/2015 11:44:32 GroupReduce (GroupReduce at > >>> main(ThiaziParser.java:40))(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > >>> > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/authors) > >>> - UTF-8)(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 CHAIN GroupReduce (GroupReduce at > >>> main(ThiaziParser.java:74)) -> Filter (Filter at > >>> main(ThiaziParser.java:97))(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > >>> cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/posts) > - > >>> UTF-8)(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 CHAIN FlatMap (FlatMap at > >>> main(ThiaziParser.java:126)) -> Combine(SUM(1), at > >>> main(ThiaziParser.java:140)(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 Reduce (SUM(1), at > >>> main(ThiaziParser.java:140)(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 DataSink(CsvOutputFormat (path: hdfs:// > >>> > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/wordcount > , > >>> delimiter: ,))(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 GroupReduce (GroupReduce at > >>> main(ThiaziParser.java:106))(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > >>> > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/threads) > >>> - UTF-8)(1/1) switched to CANCELED > >>> 01/28/2015 11:44:32 Job execution switched to status FAILED. > >>> Error: The program execution failed: java.lang.RuntimeException: > >>> Requesting the next InputSplit failed. > >>> at > >>> > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > >>> at > >>> > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > >>> at > >>> > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > >>> at > >>> > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > >>> at java.lang.Thread.run(Thread.java:745) > >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out > after > >>> [100 seconds] > >>> at > >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > >>> at > >>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > >>> at > >>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > >>> at > >>> > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > >>> at scala.concurrent.Await$.result(package.scala:107) > >>> at > >>> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > >>> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > >>> at > >>> > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > >>> ... 4 more > >>> > >>> > >>> > > > > |
I think that the machines have lost connection. That is most likely
connected to the heartbeat interval of the watch or transport failure detector. The transport failure detector should actually be set to a heartbeat interval of 1000 s and consequently it should not cause any problems. Which version of the snapshot are you exactly using Christoph? If there is still the config parameter akka.loglevel could you please set it to INFO and send us the log output again. If you use the latest snapshot, then this should no longer be necessary. Alternatively, you could try to set the akka.watch.heartbeat.interval and akka.watch.heartbeat.pause to something like 1000 s and 6000 s, respectively, and try it again. Are you running the experiments on cloud-11 Christoph? Maybe I can take a look at it if you can provide me the job jars and the Flink version. On Wed, Jan 28, 2015 at 7:57 PM, Stephan Ewen <[hidden email]> wrote: > I see the following line: > > 11:14:32,603 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system [akka.tcp:// > [hidden email]:51449] has failed, address is now gated > for [5000] ms. Reason is: [Disassociated]. > > Does that mean that the machines have lost connection? > > @Till What is your take on this? > > > On Wed, Jan 28, 2015 at 7:07 AM, Bruecke, Christoph < > [hidden email]> wrote: > > > I might add that the error only occurs when running with the > > RemoteExecutor regardless of the number of TM. Starting the job in > IntelliJ > > with the LocalExecutor with dop 1 works just fine. > > > > Best, > > Christoph > > > > On 28 Jan 2015, at 12:17, Bruecke, Christoph < > > [hidden email]> wrote: > > > > > Hi Robert, > > > > > > thanks for the quick response. Here is the jobmanager-main.log: > > > > > > PS: I’m subscribed now. > > > > > > 11:09:16,144 INFO org.apache.flink.yarn.ApplicationMaster$ > > - YARN daemon runs as hadoop setting user to execute Flink > > ApplicationMaster/JobManager to hadoop > > > 11:09:16,199 INFO org.apache.flink.yarn.Utils > > - Found YARN_CONF_DIR, adding it to configuration > > > 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ > > - Start job manager for yarn > > > 11:09:16,298 INFO org.apache.flink.yarn.ApplicationMaster$ > > - Config path: > > > /data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001. > > > 11:09:17,179 INFO akka.event.slf4j.Slf4jLogger > > - Slf4jLogger started > > > 11:09:17,271 INFO Remoting > > - Starting remoting > > > 11:09:17,556 INFO Remoting > > - Remoting started; listening on addresses :[akka.tcp:// > > [hidden email]:42643] > > > 11:09:17,569 INFO org.apache.flink.yarn.ApplicationMaster$ > > - Start job manager actor. > > > 11:09:17,570 INFO org.apache.flink.yarn.ApplicationMaster$ > > - Generate configuration file for application master. > > > 11:09:17,579 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Starting job manager at akka://flink/user/jobmanager. > > > 11:09:17,590 INFO org.apache.flink.yarn.ApplicationMaster$ > > - Start yarn session on job manager. > > > 11:09:17,591 INFO org.apache.flink.yarn.ApplicationMaster$ > > - Application Master properly initiated. Await termination of > actor > > system. > > > 11:09:17,601 INFO org.apache.flink.runtime.blob.BlobServer > > - Started BLOB server on port 35605 > > > 11:09:17,616 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Started job manager. Waiting for incoming messages. > > > 11:09:17,621 INFO > > org.apache.flink.runtime.jobmanager.web.WebInfoServer - Setting > up > > web info server, using web-root > > > directoryjar:file:/data/4/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/filecache/11/flink-dist-0.9-SNAPSHOT-yarn-uberjar.jar!/web-docs-infoserver. > > > 11:09:17,621 INFO > > org.apache.flink.runtime.jobmanager.web.WebInfoServer - Web info > > server will display information about flink job-manager on localhost, > port > > 8081. > > > 11:09:17,733 INFO > > org.apache.flink.runtime.jobmanager.web.WebInfoServer - Starting > > web info server for JobManager on port 8081 > > > 11:09:17,734 INFO org.eclipse.jetty.util.log > > - jetty-0.9-SNAPSHOT > > > 11:09:17,770 INFO org.eclipse.jetty.util.log > > - Started SelectChannelConnector@0.0.0.0:8081 > > > 11:09:17,784 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Start yarn session. > > > 11:09:17,784 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Requesting 2 task managers. > > > 11:09:18,414 INFO org.apache.hadoop.yarn.client.RMProxy > > - Connecting to ResourceManager at cloud-11/130.149.21.15:8030 > > > 11:09:18,423 INFO > > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy > - > > yarn.client.max-nodemanagers-proxies : 500 > > > 11:09:18,425 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Registering ApplicationMaster with tracking url > > http://cloud-29.dima.tu-berlin.de:8081. > > > 11:09:18,787 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Requesting TaskManager container 0. > > > 11:09:18,794 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Requesting TaskManager container 1. > > > 11:09:18,903 INFO org.apache.flink.yarn.Utils > > - Copying from > > > file:/data/3/hadoop-2.4/yarn-local-dir/usercache/hadoop/appcache/application_1420727594991_0068/container_1420727594991_0068_01_000001/flink-conf-modified.yaml > > to hdfs:// > > > cloud-11.dima.tu-berlin.de:60010/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml > > > 11:09:19,280 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Prepared local resource for modified yaml: resource { scheme: "hdfs" > > host: "cloud-11.dima.tu-berlin.de" port: 60010 file: > > > "/user/hadoop/.flink/application_1420727594991_0068/flink-conf-modified.yaml" > > } size: 3121 timestamp: 1422439759181 type: FILE visibility: APPLICATION > > > 11:09:19,286 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Create container launch context. > > > 11:09:19,300 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Starting TM with command=$JAVA_HOME/bin/java -Xmx819m > > -Dlog.file="<LOG_DIR>/taskmanager.log" > > -Dlogback.configurationFile=file:logback.xml > > -Dlog4j.configuration=file:log4j.properties > > org.apache.flink.yarn.appMaster.YarnTaskManagerRunner --configDir . 1> > > <LOG_DIR>/taskmanager-stdout.log 2> <LOG_DIR>/taskmanager-stderr.log > > > 11:09:19,619 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Got new container for TM container_1420727594991_0068_01_000002 on host > > cloud-26.dima.tu-berlin.de > > > 11:09:19,621 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Launching container #1. > > > 11:09:19,622 INFO > > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy > - > > Opening proxy : cloud-26.dima.tu-berlin.de:8045 > > > 11:09:19,828 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Got new container for TM container_1420727594991_0068_01_000003 on host > > cloud-31.dima.tu-berlin.de > > > 11:09:19,829 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Launching container #2. > > > 11:09:19,831 INFO > > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy > - > > Opening proxy : cloud-31.dima.tu-berlin.de:8045 > > > 11:09:25,748 INFO org.apache.flink.runtime.instance.InstanceManager > > - Registered TaskManager at akka.tcp:// > > [hidden email]:51449/user/taskmanager as > > ca731dbbfe941a63077c9c148254607c. Current number of registered hosts is > 1. > > > 11:09:25,845 INFO org.apache.flink.runtime.instance.InstanceManager > > - Registered TaskManager at akka.tcp:// > > [hidden email]:43200/user/taskmanager as > > 7bf3aa638e00682239c9d8460d142191. Current number of registered hosts is > 2. > > > 11:11:13,837 WARN akka.remote.ReliableDeliverySupervisor > > - Association with remote system [akka.tcp:// > > CliFrontendActorSystem@130.149.21.15:50292] has failed, address is now > > gated for [5000] ms. Reason is: [Disassociated]. > > > 11:14:30,892 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Received job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed > Jan > > 28 11:14:28 CET 2015). > > > 11:14:30,997 INFO org.apache.hadoop.conf.Configuration.deprecation > > - job.end.retry.interval is deprecated. Instead, use > > mapreduce.job.end-notification.retry.interval > > > 11:14:30,998 INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > > 11:14:30,999 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.retiredjobs.cache.size is deprecated. > Instead, > > use mapreduce.jobtracker.retiredjobs.cache.size > > > 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.profile.reduces is deprecated. Instead, use > > mapreduce.task.profile.reduces > > > 11:14:31,000 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use > > mapreduce.job.jvm.numtasks > > > 11:14:31,001 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.reduce.tasks.speculative.execution is deprecated. > Instead, > > use mapreduce.reduce.speculative > > > 11:14:31,003 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.tracker.http.address is deprecated. Instead, use > > mapreduce.tasktracker.http.address > > > 11:14:31,004 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.system.dir is deprecated. Instead, use > > mapreduce.jobtracker.system.dir > > > 11:14:31,005 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.tracker.report.address is deprecated. Instead, use > > mapreduce.tasktracker.report.address > > > 11:14:31,006 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.healthChecker.interval is deprecated. Instead, use > > mapreduce.tasktracker.healthchecker.interval > > > 11:14:31,007 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.child.tmp is deprecated. Instead, use > > mapreduce.task.tmp.dir > > > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.taskmemorymanager.monitoring-interval is > > deprecated. Instead, use > > mapreduce.tasktracker.taskmemorymanager.monitoringinterval > > > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.shuffle.connect.timeout is deprecated. Instead, use > > mapreduce.reduce.shuffle.connect.timeout > > > 11:14:31,008 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.speculative.execution.speculativeCap is deprecated. > > Instead, use mapreduce.job.speculative.speculativecap > > > 11:14:31,009 INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.sort.spill.percent is deprecated. Instead, use > > mapreduce.map.sort.spill.percent > > > 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.shuffle.input.buffer.percent is deprecated. Instead, > > use mapreduce.reduce.shuffle.input.buffer.percent > > > 11:14:31,010 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.skip.map.max.skip.records is deprecated. Instead, use > > mapreduce.map.skip.maxrecords > > > 11:14:31,011 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.profile.maps is deprecated. Instead, use > > mapreduce.task.profile.maps > > > 11:14:31,012 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.local.dir is deprecated. Instead, use > > mapreduce.cluster.local.dir > > > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.merge.recordsBeforeProgress is deprecated. Instead, use > > mapreduce.task.merge.progress.records > > > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.http.address is deprecated. Instead, use > > mapreduce.jobtracker.http.address > > > 11:14:31,013 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.userlog.retain.hours is deprecated. Instead, use > > mapreduce.job.userlog.retain.hours > > > 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.compress.map.output is deprecated. Instead, use > > mapreduce.map.output.compress > > > 11:14:31,014 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.speculative.execution.slowNodeThreshold is deprecated. > > Instead, use mapreduce.job.speculative.slownodethreshold > > > 11:14:31,015 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, > > use mapreduce.tasktracker.reduce.tasks.maximum > > > 11:14:31,017 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.reduce.child.log.level is deprecated. Instead, use > > mapreduce.reduce.log.level > > > 11:14:31,018 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.jobtracker.restart.recover is deprecated. Instead, use > > mapreduce.jobtracker.restart.recover > > > 11:14:31,020 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.inmem.merge.threshold is deprecated. Instead, use > > mapreduce.reduce.merge.inmem.threshold > > > 11:14:31,021 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.reduce.tasks is deprecated. Instead, use > > mapreduce.job.reduces > > > 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.acls.enabled is deprecated. Instead, use > > mapreduce.cluster.acls.enabled > > > 11:14:31,023 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.dns.nameserver is deprecated. Instead, use > > mapreduce.tasktracker.dns.nameserver > > > 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.submit.replication is deprecated. Instead, use > > mapreduce.client.submit.file.replication > > > 11:14:31,024 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.map.tasks.speculative.execution is deprecated. Instead, > > use mapreduce.map.speculative > > > 11:14:31,025 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.map.child.log.level is deprecated. Instead, use > > mapreduce.map.log.level > > > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.map.max.attempts is deprecated. Instead, use > > mapreduce.map.maxattempts > > > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.shuffle.merge.percent is deprecated. Instead, use > > mapreduce.reduce.shuffle.merge.percent > > > 11:14:31,026 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.jobhistory.lru.cache.size is deprecated. > > Instead, use mapreduce.jobtracker.jobhistory.lru.cache.size > > > 11:14:31,029 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.queue.name is deprecated. Instead, use > > mapreduce.job.queuename > > > 11:14:31,030 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.persist.jobstatus.hours is deprecated. > > Instead, use mapreduce.jobtracker.persist.jobstatus.hours > > > 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.healthChecker.script.timeout is deprecated. Instead, use > > mapreduce.tasktracker.healthchecker.script.timeout > > > 11:14:31,031 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.map.tasks.maximum is deprecated. Instead, > use > > mapreduce.tasktracker.map.tasks.maximum > > > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.reduce.markreset.buffer.percent is deprecated. > > Instead, use mapreduce.reduce.markreset.buffer.percent > > > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation > > - jobclient.completion.poll.interval is deprecated. Instead, use > > mapreduce.client.completion.pollinterval > > > 11:14:31,032 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.persist.jobstatus.dir is deprecated. > Instead, > > use mapreduce.jobtracker.persist.jobstatus.dir > > > 11:14:31,033 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.reduce.slowstart.completed.maps is deprecated. Instead, > > use mapreduce.job.reduce.slowstart.completedmaps > > > 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.sort.mb is deprecated. Instead, use > mapreduce.task.io.sort.mb > > > 11:14:31,034 INFO org.apache.hadoop.conf.Configuration.deprecation > > - dfs.umaskmode is deprecated. Instead, use > > fs.permissions.umask-mode > > > 11:14:31,037 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.jobtracker.instrumentation is deprecated. Instead, use > > mapreduce.jobtracker.instrumentation > > > 11:14:31,038 INFO org.apache.hadoop.conf.Configuration.deprecation > > - topology.node.switch.mapping.impl is deprecated. Instead, use > > net.topology.node.switch.mapping.impl > > > 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.output.compression.type is deprecated. Instead, use > > mapreduce.output.fileoutputformat.compress.type > > > 11:14:31,039 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.skip.attempts.to.start.skipping is deprecated. Instead, > > use mapreduce.task.skip.start.attempts > > > 11:14:31,040 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.tracker.task-controller is deprecated. Instead, use > > mapreduce.tasktracker.taskcontroller > > > 11:14:31,041 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.userlog.limit.kb is deprecated. Instead, use > > mapreduce.task.userlog.limit.kb > > > 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.committer.job.setup.cleanup.needed is deprecated. > Instead, > > use mapreduce.job.committer.setup.cleanup.needed > > > 11:14:31,043 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps > > > 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.local.dir.minspacekill is deprecated. Instead, use > > mapreduce.tasktracker.local.dir.minspacekill > > > 11:14:31,044 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapreduce.jobtracker.split.metainfo.maxsize is deprecated. > > Instead, use mapreduce.job.split.metainfo.maxsize > > > 11:14:31,045 INFO org.apache.hadoop.conf.Configuration.deprecation > > - jobclient.progress.monitor.poll.interval is deprecated. > Instead, > > use mapreduce.client.progressmonitor.pollinterval > > > 11:14:31,046 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.min.split.size is deprecated. Instead, use > > mapreduce.input.fileinputformat.split.minsize > > > 11:14:31,047 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.output.compression.codec is deprecated. Instead, use > > mapreduce.output.fileoutputformat.compress.codec > > > 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.profile is deprecated. Instead, use > > mapreduce.task.profile > > > 11:14:31,048 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.reduce.parallel.copies is deprecated. Instead, use > > mapreduce.reduce.shuffle.parallelcopies > > > 11:14:31,049 INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.sort.factor is deprecated. Instead, use > > mapreduce.task.io.sort.factor > > > 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.input.dir is deprecated. Instead, use > > mapreduce.input.fileinputformat.inputdir > > > 11:14:31,050 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.timeout is deprecated. Instead, use > > mapreduce.task.timeout > > > 11:14:31,052 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.heartbeats.in.second is deprecated. Instead, use > > mapreduce.jobtracker.heartbeats.in.second > > > 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.task.cache.levels is deprecated. Instead, use > > mapreduce.jobtracker.taskcache.levels > > > 11:14:31,053 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.map.output.compression.codec is deprecated. Instead, use > > mapreduce.map.output.compress.codec > > > 11:14:31,054 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.dns.interface is deprecated. Instead, use > > mapreduce.tasktracker.dns.interface > > > 11:14:31,055 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.max.tracker.failures is deprecated. Instead, use > > mapreduce.job.maxtaskfailures.per.tracker > > > 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation > > - dfs.df.interval is deprecated. Instead, use fs.df.interval > > > 11:14:31,056 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.tasks.sleeptime-before-sigkill is > deprecated. > > Instead, use mapreduce.tasktracker.tasks.sleeptimebeforesigkill > > > 11:14:31,057 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.max.tracker.blacklists is deprecated. Instead, use > > mapreduce.jobtracker.tasktracker.maxblacklists > > > 11:14:31,058 INFO org.apache.hadoop.conf.Configuration.deprecation > > - jobclient.output.filter is deprecated. Instead, use > > mapreduce.client.output.filter > > > 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation > > - job.end.retry.attempts is deprecated. Instead, use > > mapreduce.job.end-notification.retry.attempts > > > 11:14:31,059 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.jobtracker.taskScheduler is deprecated. Instead, use > > mapreduce.jobtracker.taskscheduler > > > 11:14:31,060 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.speculative.execution.slowTaskThreshold is deprecated. > > Instead, use mapreduce.job.speculative.slowtaskthreshold > > > 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.indexcache.mb is deprecated. Instead, use > > mapreduce.tasktracker.indexcache.mb > > > 11:14:31,061 INFO org.apache.hadoop.conf.Configuration.deprecation > > - tasktracker.http.threads is deprecated. Instead, use > > mapreduce.tasktracker.http.threads > > > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.handler.count is deprecated. Instead, use > > mapreduce.jobtracker.handler.count > > > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation > > - keep.failed.task.files is deprecated. Instead, use > > mapreduce.task.files.preserve.failedtasks > > > 11:14:31,062 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.output.compress is deprecated. Instead, use > > mapreduce.output.fileoutputformat.compress > > > 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.jobtracker.job.history.block.size is deprecated. > Instead, > > use mapreduce.jobtracker.jobhistory.block.size > > > 11:14:31,063 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.skip.reduce.max.skip.groups is deprecated. Instead, use > > mapreduce.reduce.skip.maxgroups > > > 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation > > - topology.script.number.args is deprecated. Instead, use > > net.topology.script.number.args > > > 11:14:31,066 INFO org.apache.hadoop.conf.Configuration.deprecation > > - fs.default.name is deprecated. Instead, use fs.defaultFS > > > 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.local.dir.minspacestart is deprecated. Instead, use > > mapreduce.tasktracker.local.dir.minspacestart > > > 11:14:31,067 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use > > mapreduce.jobtracker.maxtasks.perjob > > > 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.reduce.max.attempts is deprecated. Instead, use > > mapreduce.reduce.maxattempts > > > 11:14:31,068 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker is deprecated. Instead, use > > mapreduce.jobtracker.address > > > 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.shuffle.read.timeout is deprecated. Instead, use > > mapreduce.reduce.shuffle.read.timeout > > > 11:14:31,069 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.instrumentation is deprecated. Instead, use > > mapreduce.tasktracker.instrumentation > > > 11:14:31,070 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.tasktracker.expiry.interval is deprecated. Instead, use > > mapreduce.jobtracker.expire.trackers.interval > > > 11:14:31,071 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.tracker.persist.jobstatus.active is deprecated. > > Instead, use mapreduce.jobtracker.persist.jobstatus.active > > > 11:14:31,074 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.temp.dir is deprecated. Instead, use > > mapreduce.cluster.temp.dir > > > 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation > > - hadoop.native.lib is deprecated. Instead, use > > io.native.lib.available > > > 11:14:31,075 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapreduce.job.counters.limit is deprecated. Instead, use > > mapreduce.job.counters.max > > > 11:14:31,076 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapred.job.reduce.input.buffer.percent is deprecated. Instead, > > use mapreduce.reduce.input.buffer.percent > > > 11:14:31,149 INFO > > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total > input > > paths to process : 1 > > > 11:14:31,177 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Scheduling job Flink Java Job at Wed Jan 28 11:14:28 CET 2015. > > > 11:14:31,188 INFO > > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) > > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > > FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to > > cloud-26 > > > 11:14:31,202 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed > Jan > > 28 11:14:28 CET 2015) changed to RUNNING. > > > 11:14:32,525 INFO > > org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning > > remote split to host cloud-26 > > > 11:14:32,603 WARN akka.remote.ReliableDeliverySupervisor > > - Association with remote system [akka.tcp:// > > [hidden email]:51449] has failed, address is now gated > > for [5000] ms. Reason is: [Disassociated]. > > > 11:16:12,574 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed > Jan > > 28 11:14:28 CET 2015) changed to FAILINGjava.lang.RuntimeException: > > Requesting the next InputSplit failed. > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > > at > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after [100 seconds] > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > > ... 4 more > > > . > > > 11:16:12,584 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job c979ea0b2e358fb988f5a2a30ab48786 (Flink Java Job at Wed > Jan > > 28 11:14:28 CET 2015) changed to FAILEDjava.lang.RuntimeException: > > Requesting the next InputSplit failed. > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > > at > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after [100 seconds] > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > > ... 4 more > > > . > > > 11:17:53,055 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Received job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed > Jan > > 28 11:17:50 CET 2015). > > > 11:17:53,258 INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > > 11:17:53,312 INFO org.apache.hadoop.conf.Configuration.deprecation > > - fs.default.name is deprecated. Instead, use fs.defaultFS > > > 11:17:53,320 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapreduce.job.counters.limit is deprecated. Instead, use > > mapreduce.job.counters.max > > > 11:17:53,347 INFO > > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total > input > > paths to process : 1 > > > 11:17:53,352 INFO > > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) > > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > > FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to > > cloud-31 > > > 11:17:53,353 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Scheduling job Flink Java Job at Wed Jan 28 11:17:50 CET 2015. > > > 11:17:53,354 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed > Jan > > 28 11:17:50 CET 2015) changed to RUNNING. > > > 11:17:54,646 INFO > > org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning > > remote split to host cloud-31 > > > 11:17:54,715 WARN akka.remote.ReliableDeliverySupervisor > > - Association with remote system [akka.tcp:// > > [hidden email]:43200] has failed, address is now gated > > for [5000] ms. Reason is: [Disassociated]. > > > 11:19:34,684 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed > Jan > > 28 11:17:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: > > Requesting the next InputSplit failed. > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > > at > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after [100 seconds] > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > > ... 4 more > > > . > > > 11:19:34,697 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job fe8cebf0874c7f67ac03454de8555fb9 (Flink Java Job at Wed > Jan > > 28 11:17:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: > > Requesting the next InputSplit failed. > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > > at > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after [100 seconds] > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > > ... 4 more > > > . > > > 11:42:52,441 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Received job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed > Jan > > 28 11:42:50 CET 2015). > > > 11:42:52,650 INFO org.apache.hadoop.conf.Configuration.deprecation > > - io.bytes.per.checksum is deprecated. Instead, use > > dfs.bytes-per-checksum > > > 11:42:52,705 INFO org.apache.hadoop.conf.Configuration.deprecation > > - fs.default.name is deprecated. Instead, use fs.defaultFS > > > 11:42:52,713 INFO org.apache.hadoop.conf.Configuration.deprecation > > - mapreduce.job.counters.limit is deprecated. Instead, use > > mapreduce.job.counters.max > > > 11:42:52,740 INFO > > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total > input > > paths to process : 1 > > > 11:42:52,747 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Scheduling job Flink Java Job at Wed Jan 28 11:42:50 CET 2015. > > > 11:42:52,747 INFO > > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > CHAIN DataSource (at createInput(ExecutionEnvironment.java:426) > > (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) -> > > FlatMap (FlatMap at main(ThiaziParser.java:37)) (1/1) (attempt #0) to > > cloud-26 > > > 11:42:52,750 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed > Jan > > 28 11:42:50 CET 2015) changed to RUNNING. > > > 11:42:52,956 INFO > > org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning > > remote split to host cloud-26 > > > 11:42:52,983 WARN akka.remote.ReliableDeliverySupervisor > > - Association with remote system [akka.tcp:// > > [hidden email]:51449] has failed, address is now gated > > for [5000] ms. Reason is: [Disassociated]. > > > 11:44:32,976 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed > Jan > > 28 11:42:50 CET 2015) changed to FAILINGjava.lang.RuntimeException: > > Requesting the next InputSplit failed. > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > > at > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after [100 seconds] > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > > ... 4 more > > > . > > > 11:44:32,984 INFO > > > org.apache.flink.yarn.ApplicationMaster$$anonfun$startJobManager$2$$anon$1 > > - Status of job b63a600c2edd93b790f2e984ca987464 (Flink Java Job at Wed > Jan > > 28 11:42:50 CET 2015) changed to FAILEDjava.lang.RuntimeException: > > Requesting the next InputSplit failed. > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > > at > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > > at > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after [100 seconds] > > > at > > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > > at > > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > > at > > scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > > at > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > > at scala.concurrent.Await$.result(package.scala:107) > > > at > > org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > > at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > > at > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > > ... 4 more > > > . > > > > > > On 28 Jan 2015, at 12:08, Robert Metzger <[hidden email]> wrote: > > > > > >> Hi, > > >> > > >> it seems that you are not subscribed to our mailing list, so I had to > > >> manually accept your mail. Would be good if you could subscribe. > > >> > > >> Can you send us also the log output of the JobManager? > > >> If your YARN cluster has log aggregation activated, you can retrieve > the > > >> logs of a stopped YARN session using: > > >> yarn logs -applicationId <AppId> > > >> > > >> watch out for the jobmanager-main.log or so file. > > >> > > >> I suspect that there has been an exception on the JobManager. > > >> > > >> Best, > > >> Robert > > >> > > >> > > >> > > >> On Wed, Jan 28, 2015 at 12:01 PM, Bruecke, Christoph < > > >> [hidden email]> wrote: > > >> > > >>> Hi, > > >>> > > >>> I have written a job that reads a SequenceFile from HDFS using the > > >>> Hadoop-Compatibility add-on. Doing so results in a TimeoutException. > > I’m > > >>> using flink-0.9-SNAPSHOT with PR 342 ( > > >>> https://github.com/apache/flink/pull/342 ). Furthermore I’m running > > flink > > >>> on yarn with two TM using flink-yarn-0.9-SNAPSHOT/bin/yarn-session.sh > > -n 2. > > >>> > > >>> Is this a bug or is there something wrong with the configuration? > > >>> > > >>> 01/28/2015 11:42:52 Job execution switched to status RUNNING. > > >>> 01/28/2015 11:42:52 CHAIN DataSource (at > > >>> createInput(ExecutionEnvironment.java:426) > > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) > -> > > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > > SCHEDULED > > >>> 01/28/2015 11:42:52 CHAIN DataSource (at > > >>> createInput(ExecutionEnvironment.java:426) > > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) > -> > > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > > DEPLOYING > > >>> 01/28/2015 11:42:52 CHAIN DataSource (at > > >>> createInput(ExecutionEnvironment.java:426) > > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) > -> > > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > > RUNNING > > >>> 01/28/2015 11:44:32 CHAIN DataSource (at > > >>> createInput(ExecutionEnvironment.java:426) > > >>> (org.apache.flink.hadoopcompatibility.mapreduce.HadoopInputFormat)) > -> > > >>> FlatMap (FlatMap at main(ThiaziParser.java:37))(1/1) switched to > FAILED > > >>> java.lang.RuntimeException: Requesting the next InputSplit failed. > > >>> at > > >>> > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > >>> at > > >>> > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > >>> at > > >>> > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > >>> at > > >>> > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > >>> at java.lang.Thread.run(Thread.java:745) > > >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after > > >>> [100 seconds] > > >>> at > > >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > >>> at > > >>> > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > >>> at > > >>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > >>> at > > >>> > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > >>> at scala.concurrent.Await$.result(package.scala:107) > > >>> at > > >>> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > >>> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > >>> at > > >>> > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > >>> ... 4 more > > >>> > > >>> 01/28/2015 11:44:32 Job execution switched to status FAILING. > > >>> 01/28/2015 11:44:32 GroupReduce (GroupReduce at > > >>> main(ThiaziParser.java:40))(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > > >>> > > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/authors > ) > > >>> - UTF-8)(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 CHAIN GroupReduce (GroupReduce at > > >>> main(ThiaziParser.java:74)) -> Filter (Filter at > > >>> main(ThiaziParser.java:97))(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > > >>> > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/posts) > > - > > >>> UTF-8)(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 CHAIN FlatMap (FlatMap at > > >>> main(ThiaziParser.java:126)) -> Combine(SUM(1), at > > >>> main(ThiaziParser.java:140)(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 Reduce (SUM(1), at > > >>> main(ThiaziParser.java:140)(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 DataSink(CsvOutputFormat (path: hdfs:// > > >>> > > > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/wordcount > > , > > >>> delimiter: ,))(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 GroupReduce (GroupReduce at > > >>> main(ThiaziParser.java:106))(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 DataSink(TextOutputFormat (hdfs:// > > >>> > > cloud-11.dima.tu-berlin.de:60010/user/cbruecke/output/thiazi-seq/threads > ) > > >>> - UTF-8)(1/1) switched to CANCELED > > >>> 01/28/2015 11:44:32 Job execution switched to status FAILED. > > >>> Error: The program execution failed: java.lang.RuntimeException: > > >>> Requesting the next InputSplit failed. > > >>> at > > >>> > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:63) > > >>> at > > >>> > > > org.apache.flink.runtime.operators.DataSourceTask$1.hasNext(DataSourceTask.java:355) > > >>> at > > >>> > > > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:154) > > >>> at > > >>> > > > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) > > >>> at java.lang.Thread.run(Thread.java:745) > > >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out > > after > > >>> [100 seconds] > > >>> at > > >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > >>> at > > >>> > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > >>> at > > >>> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > >>> at > > >>> > > > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > >>> at scala.concurrent.Await$.result(package.scala:107) > > >>> at > > >>> org.apache.flink.runtime.akka.AkkaUtils$.ask(AkkaUtils.scala:265) > > >>> at org.apache.flink.runtime.akka.AkkaUtils.ask(AkkaUtils.scala) > > >>> at > > >>> > > > org.apache.flink.runtime.taskmanager.TaskInputSplitProvider.getNextInputSplit(TaskInputSplitProvider.java:56) > > >>> ... 4 more > > >>> > > >>> > > >>> > > > > > > > > |
@Till: The default timeouts are high enough that such a timeout should
actually not occur, right? Increasing the timeouts cannot really be the issue. Might it be something different? What happens if there is an error in the code that produces the input split? Is that properly handled, or is the receiver simply not ever getting an answer. Could that be what happened? |
Yes actually the timeouts should not really matter. However, an exception
in the InputSplitAssigner should happen in the actor thread and thus cause the actor to stop. This should be logged by the supervisor. I just checked and the method InputSplitAssigner.getNextInputSplit is not supposed to throw any checked exceptions. On Thu, Jan 29, 2015 at 6:38 AM, Stephan Ewen <[hidden email]> wrote: > @Till: The default timeouts are high enough that such a timeout should > actually not occur, right? Increasing the timeouts cannot really be the > issue. > > Might it be something different? What happens if there is an error in the > code that produces the input split? Is that properly handled, or is the > receiver simply not ever getting an answer. Could that be what happened? > |
@Till: Yes I’m running the job on cloud-11 or better to say I’m using the yarn cluster and the flink-yarn package. I’m using flink-0.9-SNAPSHOT from the following commit [1] together with Timos patch [2]. I’ll send you a separate email with instructions where you can find the jars on cloud-11.
Thanks, Christoph [1] https://github.com/apache/flink/commit/06b2acf7946efc03760cacf464cead5ec2d8daa6 [2] https://github.com/apache/flink/pull/342 On 29 Jan 2015, at 09:14, Till Rohrmann <[hidden email]> wrote: > Yes actually the timeouts should not really matter. However, an exception > in the InputSplitAssigner should happen in the actor thread and thus cause > the actor to stop. This should be logged by the supervisor. > > I just checked and the method InputSplitAssigner.getNextInputSplit is not > supposed to throw any checked exceptions. > > On Thu, Jan 29, 2015 at 6:38 AM, Stephan Ewen <[hidden email]> wrote: > >> @Till: The default timeouts are high enough that such a timeout should >> actually not occur, right? Increasing the timeouts cannot really be the >> issue. >> >> Might it be something different? What happens if there is an error in the >> code that produces the input split? Is that properly handled, or is the >> receiver simply not ever getting an answer. Could that be what happened? >> |
In reply to this post by Till Rohrmann
I looked into the problem and the problem is a deserialization issue on the
TaskManager side. Somehow the system is not capable to send InputSplits around whose classes are contained in the user code jars. A similar issue was already observed by Fabian in FLINK-1438. I used his test program and the same error as with Thiazi-parser occurred. Furthermore, I could solve the problem by putting the flink-hadoop-compatibility jar in the lib folder of Flink. This is the same behavior as Fabian described. On Thu, Jan 29, 2015 at 9:14 AM, Till Rohrmann <[hidden email]> wrote: > Yes actually the timeouts should not really matter. However, an exception > in the InputSplitAssigner should happen in the actor thread and thus cause > the actor to stop. This should be logged by the supervisor. > > I just checked and the method InputSplitAssigner.getNextInputSplit is not > supposed to throw any checked exceptions. > > On Thu, Jan 29, 2015 at 6:38 AM, Stephan Ewen <[hidden email]> wrote: > >> @Till: The default timeouts are high enough that such a timeout should >> actually not occur, right? Increasing the timeouts cannot really be the >> issue. >> >> Might it be something different? What happens if there is an error in the >> code that produces the input split? Is that properly handled, or is the >> receiver simply not ever getting an answer. Could that be what happened? >> > > |
I've updated the corresponding jira ticket.
On Fri, Jan 30, 2015 at 5:46 PM, Till Rohrmann <[hidden email]> wrote: > I looked into the problem and the problem is a deserialization issue on > the TaskManager side. Somehow the system is not capable to send InputSplits > around whose classes are contained in the user code jars. A similar issue > was already observed by Fabian in FLINK-1438. I used his test program and > the same error as with Thiazi-parser occurred. Furthermore, I could solve > the problem by putting the flink-hadoop-compatibility jar in the lib folder > of Flink. This is the same behavior as Fabian described. > > On Thu, Jan 29, 2015 at 9:14 AM, Till Rohrmann <[hidden email]> > wrote: > >> Yes actually the timeouts should not really matter. However, an exception >> in the InputSplitAssigner should happen in the actor thread and thus cause >> the actor to stop. This should be logged by the supervisor. >> >> I just checked and the method InputSplitAssigner.getNextInputSplit is not >> supposed to throw any checked exceptions. >> >> On Thu, Jan 29, 2015 at 6:38 AM, Stephan Ewen <[hidden email]> wrote: >> >>> @Till: The default timeouts are high enough that such a timeout should >>> actually not occur, right? Increasing the timeouts cannot really be the >>> issue. >>> >>> Might it be something different? What happens if there is an error in the >>> code that produces the input split? Is that properly handled, or is the >>> receiver simply not ever getting an answer. Could that be what happened? >>> >> >> > |
Free forum by Nabble | Edit this page |