Hello,
I am about to deploy my first Flink projects to production, but I am running into a very big hurdle. I am unable to launch my project so it can write to an S3 bucket. My project is running on an EMR cluster, where I have installed Flink 1.3.2. I am using Yarn to launch the application, and it seems to run fine unless I am trying to enable check pointing(with a S3 target). I am looking to use RocksDB as my check-pointing backend. I have asked a few places, and I am still unable to find a solution to this problem. Here are my steps for creating a cluster, and launching my application, perhaps I am missing a step. I'd be happy to provide any additional information if needed. AWS Portal: 1) EMR -> Create Cluster 2) Advanced Options 3) Release = emr-5.8.0 4) Only select Hadoop 2.7.3 5) Next -> Next -> Next -> Create Cluster ( I do fill out names/keys/etc) Once the cluster is up I ssh into the Master and do the following: 1 wget http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin-hadoop27-scala_2.11.tgz 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz 3 cd flink-1.3.2 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d 5 Change conf/flink-conf.yaml 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar My conf/flink-conf.yaml I add the following fields: state.backend: rocksdb state.backend.fs.checkpointdir: s3:/bucket/location state.checkpoints.dir: s3:/bucket/location My program's checkpointing setup: env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE) env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) env.getCheckpointConfig.setMinPauseBetweenCheckpoints(getCheckpointMinPause) env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout) env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) env.setStateBackend(new RocksDBStateBackend("s3://bucket/location", true)) |
Hi Andy,
can you describe in more detail what exactly isn't working? Do you see error messages in the log files or on the console? Thanks, Fabian 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > Hello, > > I am about to deploy my first Flink projects to production, but I am > running into a very big hurdle. I am unable to launch my project so it can > write to an S3 bucket. My project is running on an EMR cluster, where I > have installed Flink 1.3.2. I am using Yarn to launch the application, and > it seems to run fine unless I am trying to enable check pointing(with a S3 > target). I am looking to use RocksDB as my check-pointing backend. I have > asked a few places, and I am still unable to find a solution to this > problem. Here are my steps for creating a cluster, and launching my > application, perhaps I am missing a step. I'd be happy to provide any > additional information if needed. > > AWS Portal: > > 1) EMR -> Create Cluster > 2) Advanced Options > 3) Release = emr-5.8.0 > 4) Only select Hadoop 2.7.3 > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > names/keys/etc) > > Once the cluster is up I ssh into the Master and do the following: > > 1 wget > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > hadoop27-scala_2.11.tgz > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > 3 cd flink-1.3.2 > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > 5 Change conf/flink-conf.yaml > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar > > My conf/flink-conf.yaml I add the following fields: > > state.backend: rocksdb > state.backend.fs.checkpointdir: s3:/bucket/location > state.checkpoints.dir: s3:/bucket/location > > My program's checkpointing setup: > > > env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE) > > env.getCheckpointConfig.enableExternalizedCheckpoints( > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > getCheckpointMinPause) > env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout) > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > env.setStateBackend(new RocksDBStateBackend("s3://bucket/location", > true)) > |
Hi Fabian,
Sorry, I just realized I forgot to include that part. The error returned is: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:93) at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:328) at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:350) at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:282) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273 I believe it has something to do with the classpath, but I am unsure why or how to fix it. The classpath being used during the execution is: /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems the addResource function does seem to be there. Thank you On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email]> wrote: > Hi Andy, > > can you describe in more detail what exactly isn't working? > Do you see error messages in the log files or on the console? > > Thanks, Fabian > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > Hello, > > > > I am about to deploy my first Flink projects to production, but I am > > running into a very big hurdle. I am unable to launch my project so it > can > > write to an S3 bucket. My project is running on an EMR cluster, where I > > have installed Flink 1.3.2. I am using Yarn to launch the application, > and > > it seems to run fine unless I am trying to enable check pointing(with a > S3 > > target). I am looking to use RocksDB as my check-pointing backend. I > have > > asked a few places, and I am still unable to find a solution to this > > problem. Here are my steps for creating a cluster, and launching my > > application, perhaps I am missing a step. I'd be happy to provide any > > additional information if needed. > > > > AWS Portal: > > > > 1) EMR -> Create Cluster > > 2) Advanced Options > > 3) Release = emr-5.8.0 > > 4) Only select Hadoop 2.7.3 > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > names/keys/etc) > > > > Once the cluster is up I ssh into the Master and do the following: > > > > 1 wget > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > hadoop27-scala_2.11.tgz > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > 3 cd flink-1.3.2 > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > 5 Change conf/flink-conf.yaml > > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar > > > > My conf/flink-conf.yaml I add the following fields: > > > > state.backend: rocksdb > > state.backend.fs.checkpointdir: s3:/bucket/location > > state.checkpoints.dir: s3:/bucket/location > > > > My program's checkpointing setup: > > > > > > env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ > ONCE) > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > getCheckpointMinPause) > > env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout) > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > env.setStateBackend(new RocksDBStateBackend("s3://bucket/location", > > true)) > > > |
Hi Andy,
I'm not an AWS expert, so I'll just check on some common issues. I guess you already had a look at the Flink docs for AWS/EMR but I'll post the link just be to sure [1]. Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did you built Flink yourself or did you download the binaries? Does the Hadoop version of the Flink build match the Hadoop version of EMR 5.8.0, i.e., Hadoop 2.7.x? Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > Hi Fabian, > > Sorry, I just realized I forgot to include that part. The error returned > is: > > java.lang.NoSuchMethodError: > org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/ > Configuration;)V > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > EmrFileSystem.java:93) > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > initialize(HadoopFileSystem.java:328) > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > FileSystem.java:350) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > at org.apache.flink.runtime.state.filesystem. > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > createStreamFactory(FsStateBackend.java:282) > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend. > createStreamFactory(RocksDBStateBackend.java:273 > > I believe it has something to do with the classpath, but I am unsure why or > how to fix it. The classpath being used during the execution is: > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > 2.jar::/etc/hadoop/conf: > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems the > addResource function does seem to be there. > > Thank you > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email]> wrote: > > > Hi Andy, > > > > can you describe in more detail what exactly isn't working? > > Do you see error messages in the log files or on the console? > > > > Thanks, Fabian > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > Hello, > > > > > > I am about to deploy my first Flink projects to production, but I am > > > running into a very big hurdle. I am unable to launch my project so it > > can > > > write to an S3 bucket. My project is running on an EMR cluster, where > I > > > have installed Flink 1.3.2. I am using Yarn to launch the application, > > and > > > it seems to run fine unless I am trying to enable check pointing(with a > > S3 > > > target). I am looking to use RocksDB as my check-pointing backend. I > > have > > > asked a few places, and I am still unable to find a solution to this > > > problem. Here are my steps for creating a cluster, and launching my > > > application, perhaps I am missing a step. I'd be happy to provide any > > > additional information if needed. > > > > > > AWS Portal: > > > > > > 1) EMR -> Create Cluster > > > 2) Advanced Options > > > 3) Release = emr-5.8.0 > > > 4) Only select Hadoop 2.7.3 > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > names/keys/etc) > > > > > > Once the cluster is up I ssh into the Master and do the following: > > > > > > 1 wget > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > hadoop27-scala_2.11.tgz > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > 3 cd flink-1.3.2 > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > 5 Change conf/flink-conf.yaml > > > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > state.backend: rocksdb > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > state.checkpoints.dir: s3:/bucket/location > > > > > > My program's checkpointing setup: > > > > > > > > > env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ > > ONCE) > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > getCheckpointMinPause) > > > env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout) > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > env.setStateBackend(new RocksDBStateBackend("s3:// > bucket/location", > > > true)) > > > > > > |
Hi Fabian,
1) I have looked at the linked docs, and from what I can tell no setup should really need to be done to get Flink working(Other than downloading the correct binaries, which I believe I did) 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- hadoop27-scala_2.11.tgz <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin-hadoop27-scala_2.11.tgz>) This is for hadoop 2.7.X, which matches EMR 5.8.0. I appreciate any help or guidance you can provide me in fixing my problems, please let me know if there is anything else I can provide you. Thank you On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email]> wrote: > Hi Andy, > > I'm not an AWS expert, so I'll just check on some common issues. > > I guess you already had a look at the Flink docs for AWS/EMR but I'll post > the link just be to sure [1]. > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did you > built Flink yourself or did you download the binaries? > Does the Hadoop version of the Flink build match the Hadoop version of EMR > 5.8.0, i.e., Hadoop 2.7.x? > > Best, Fabian > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > Hi Fabian, > > > > Sorry, I just realized I forgot to include that part. The error returned > > is: > > > > java.lang.NoSuchMethodError: > > org.apache.hadoop.conf.Configuration.addResource( > Lorg/apache/hadoop/conf/ > > Configuration;)V > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > EmrFileSystem.java:93) > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > initialize(HadoopFileSystem.java:328) > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > > FileSystem.java:350) > > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > > at org.apache.flink.runtime.state.filesystem. > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > createStreamFactory(FsStateBackend.java:282) > > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend. > > createStreamFactory(RocksDBStateBackend.java:273 > > > > I believe it has something to do with the classpath, but I am unsure why > or > > how to fix it. The classpath being used during the execution is: > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > 2.jar::/etc/hadoop/conf: > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems the > > addResource function does seem to be there. > > > > Thank you > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email]> wrote: > > > > > Hi Andy, > > > > > > can you describe in more detail what exactly isn't working? > > > Do you see error messages in the log files or on the console? > > > > > > Thanks, Fabian > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > Hello, > > > > > > > > I am about to deploy my first Flink projects to production, but I am > > > > running into a very big hurdle. I am unable to launch my project so > it > > > can > > > > write to an S3 bucket. My project is running on an EMR cluster, > where > > I > > > > have installed Flink 1.3.2. I am using Yarn to launch the > application, > > > and > > > > it seems to run fine unless I am trying to enable check > pointing(with a > > > S3 > > > > target). I am looking to use RocksDB as my check-pointing backend. > I > > > have > > > > asked a few places, and I am still unable to find a solution to this > > > > problem. Here are my steps for creating a cluster, and launching my > > > > application, perhaps I am missing a step. I'd be happy to provide > any > > > > additional information if needed. > > > > > > > > AWS Portal: > > > > > > > > 1) EMR -> Create Cluster > > > > 2) Advanced Options > > > > 3) Release = emr-5.8.0 > > > > 4) Only select Hadoop 2.7.3 > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > > names/keys/etc) > > > > > > > > Once the cluster is up I ssh into the Master and do the following: > > > > > > > > 1 wget > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > hadoop27-scala_2.11.tgz > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > 3 cd flink-1.3.2 > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > 5 Change conf/flink-conf.yaml > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > state.backend: rocksdb > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ > > > ONCE) > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > getCheckpointMinPause) > > > > env.getCheckpointConfig.setCheckpointTimeout( > getCheckpointTimeout) > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > bucket/location", > > > > true)) > > > > > > > > > > |
Hi Andy,
this looks to me indeed like a dependency problem. I assume that EMR or something else is pulling in an incompatible version of Hadoop. The classpath you've posted, is this the one logged in the log files (TaskManager log) or did you compile it yourself? In the latter case, it would also be helpful to get access to the TaskManager logs. Cheers, Till On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> wrote: > Hi Fabian, > > 1) I have looked at the linked docs, and from what I can tell no setup > should really need to be done to get Flink working(Other than downloading > the correct binaries, which I believe I did) > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > hadoop27-scala_2.11.tgz > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > hadoop27-scala_2.11.tgz>) > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > I appreciate any help or guidance you can provide me in fixing my problems, > please let me know if there is anything else I can provide you. > > Thank you > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email]> wrote: > > > Hi Andy, > > > > I'm not an AWS expert, so I'll just check on some common issues. > > > > I guess you already had a look at the Flink docs for AWS/EMR but I'll > post > > the link just be to sure [1]. > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did > you > > built Flink yourself or did you download the binaries? > > Does the Hadoop version of the Flink build match the Hadoop version of > EMR > > 5.8.0, i.e., Hadoop 2.7.x? > > > > Best, Fabian > > > > [1] > > https://ci.apache.org/projects/flink/flink-docs- > release-1.3/setup/aws.html > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > Hi Fabian, > > > > > > Sorry, I just realized I forgot to include that part. The error > returned > > > is: > > > > > > java.lang.NoSuchMethodError: > > > org.apache.hadoop.conf.Configuration.addResource( > > Lorg/apache/hadoop/conf/ > > > Configuration;)V > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > EmrFileSystem.java:93) > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > initialize(HadoopFileSystem.java:328) > > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > > > FileSystem.java:350) > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > > > at org.apache.flink.runtime.state.filesystem. > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) > > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > > createStreamFactory(FsStateBackend.java:282) > > > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend. > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > I believe it has something to do with the classpath, but I am unsure > why > > or > > > how to fix it. The classpath being used during the execution is: > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > 2.jar::/etc/hadoop/conf: > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems the > > > addResource function does seem to be there. > > > > > > Thank you > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email]> > wrote: > > > > > > > Hi Andy, > > > > > > > > can you describe in more detail what exactly isn't working? > > > > Do you see error messages in the log files or on the console? > > > > > > > > Thanks, Fabian > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > Hello, > > > > > > > > > > I am about to deploy my first Flink projects to production, but I > am > > > > > running into a very big hurdle. I am unable to launch my project > so > > it > > > > can > > > > > write to an S3 bucket. My project is running on an EMR cluster, > > where > > > I > > > > > have installed Flink 1.3.2. I am using Yarn to launch the > > application, > > > > and > > > > > it seems to run fine unless I am trying to enable check > > pointing(with a > > > > S3 > > > > > target). I am looking to use RocksDB as my check-pointing backend. > > I > > > > have > > > > > asked a few places, and I am still unable to find a solution to > this > > > > > problem. Here are my steps for creating a cluster, and launching > my > > > > > application, perhaps I am missing a step. I'd be happy to provide > > any > > > > > additional information if needed. > > > > > > > > > > AWS Portal: > > > > > > > > > > 1) EMR -> Create Cluster > > > > > 2) Advanced Options > > > > > 3) Release = emr-5.8.0 > > > > > 4) Only select Hadoop 2.7.3 > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > > > names/keys/etc) > > > > > > > > > > Once the cluster is up I ssh into the Master and do the following: > > > > > > > > > > 1 wget > > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > hadoop27-scala_2.11.tgz > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > 3 cd flink-1.3.2 > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > 5 Change conf/flink-conf.yaml > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > state.backend: rocksdb > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > CheckpointingMode.EXACTLY_ > > > > ONCE) > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > getCheckpointMinPause) > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > getCheckpointTimeout) > > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > bucket/location", > > > > > true)) > > > > > > > > > > > > > > > |
Hi Till,
That is actually the classpath used by the flink bash script(that launches the jar using the java command). I changed the execute to an echo, and grabbed that for the CLI arguments. I believe this is the class path from the log file(although it might not be the taskmanager log, is that any different from what would be in my flink-1.3.2/log folder?): 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend - Classpath: /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: If that doesn't seem right, and you can point me in the right direction as to where the TaskManager logs would be, I would be happy to grab the information your looking for. Thank you On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <[hidden email]> wrote: > Hi Andy, > > this looks to me indeed like a dependency problem. I assume that EMR or > something else is pulling in an incompatible version of Hadoop. > > The classpath you've posted, is this the one logged in the log files > (TaskManager log) or did you compile it yourself? In the latter case, it > would also be helpful to get access to the TaskManager logs. > > Cheers, > Till > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> wrote: > > > Hi Fabian, > > > > 1) I have looked at the linked docs, and from what I can tell no setup > > should really need to be done to get Flink working(Other than downloading > > the correct binaries, which I believe I did) > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > hadoop27-scala_2.11.tgz > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > hadoop27-scala_2.11.tgz>) > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > I appreciate any help or guidance you can provide me in fixing my > problems, > > please let me know if there is anything else I can provide you. > > > > Thank you > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email]> wrote: > > > > > Hi Andy, > > > > > > I'm not an AWS expert, so I'll just check on some common issues. > > > > > > I guess you already had a look at the Flink docs for AWS/EMR but I'll > > post > > > the link just be to sure [1]. > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did > > you > > > built Flink yourself or did you download the binaries? > > > Does the Hadoop version of the Flink build match the Hadoop version of > > EMR > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > Best, Fabian > > > > > > [1] > > > https://ci.apache.org/projects/flink/flink-docs- > > release-1.3/setup/aws.html > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > Hi Fabian, > > > > > > > > Sorry, I just realized I forgot to include that part. The error > > returned > > > > is: > > > > > > > > java.lang.NoSuchMethodError: > > > > org.apache.hadoop.conf.Configuration.addResource( > > > Lorg/apache/hadoop/conf/ > > > > Configuration;)V > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > > EmrFileSystem.java:93) > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > initialize(HadoopFileSystem.java:328) > > > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > > > > FileSystem.java:350) > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > > > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > > > > at org.apache.flink.runtime.state.filesystem. > > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99) > > > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > > > createStreamFactory(FsStateBackend.java:282) > > > > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend. > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > I believe it has something to do with the classpath, but I am unsure > > why > > > or > > > > how to fix it. The classpath being used during the execution is: > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems > the > > > > addResource function does seem to be there. > > > > > > > > Thank you > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email]> > > wrote: > > > > > > > > > Hi Andy, > > > > > > > > > > can you describe in more detail what exactly isn't working? > > > > > Do you see error messages in the log files or on the console? > > > > > > > > > > Thanks, Fabian > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > Hello, > > > > > > > > > > > > I am about to deploy my first Flink projects to production, but > I > > am > > > > > > running into a very big hurdle. I am unable to launch my project > > so > > > it > > > > > can > > > > > > write to an S3 bucket. My project is running on an EMR cluster, > > > where > > > > I > > > > > > have installed Flink 1.3.2. I am using Yarn to launch the > > > application, > > > > > and > > > > > > it seems to run fine unless I am trying to enable check > > > pointing(with a > > > > > S3 > > > > > > target). I am looking to use RocksDB as my check-pointing > backend. > > > I > > > > > have > > > > > > asked a few places, and I am still unable to find a solution to > > this > > > > > > problem. Here are my steps for creating a cluster, and launching > > my > > > > > > application, perhaps I am missing a step. I'd be happy to > provide > > > any > > > > > > additional information if needed. > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > 2) Advanced Options > > > > > > 3) Release = emr-5.8.0 > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > > > > names/keys/etc) > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > following: > > > > > > > > > > > > 1 wget > > > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > hadoop27-scala_2.11.tgz > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > > 3 cd flink-1.3.2 > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > 5 Change conf/flink-conf.yaml > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > state.backend: rocksdb > > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > CheckpointingMode.EXACTLY_ > > > > > ONCE) > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > getCheckpointMinPause) > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > getCheckpointTimeout) > > > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > bucket/location", > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > |
Hi Andy,
the CliFrontend is not executed via Yarn, thus, it is not affected by dependencies which are added due to the underlying Yarn cluster. Therefore, it would be helpful to look at the TaskManager logs. Either you have enabled log aggregation on your Yarn cluster, then you can obtain the logs via `yarn logs -applicationId <application ID>` or you have to retrieve them from the machines where they were running (either by going directly there or via the Yarn web interface). Cheers, Till On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> wrote: > Hi Till, > > That is actually the classpath used by the flink bash script(that launches > the jar using the java command). I changed the execute to an echo, and > grabbed that for the CLI arguments. > > I believe this is the class path from the log file(although it might not be > the taskmanager log, is that any different from what would be in my > flink-1.3.2/log folder?): > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend > - Classpath: > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > > If that doesn't seem right, and you can point me in the right direction as > to where the TaskManager logs would be, I would be happy to grab the > information your looking for. > > Thank you > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Andy, > > > > this looks to me indeed like a dependency problem. I assume that EMR or > > something else is pulling in an incompatible version of Hadoop. > > > > The classpath you've posted, is this the one logged in the log files > > (TaskManager log) or did you compile it yourself? In the latter case, it > > would also be helpful to get access to the TaskManager logs. > > > > Cheers, > > Till > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> wrote: > > > > > Hi Fabian, > > > > > > 1) I have looked at the linked docs, and from what I can tell no setup > > > should really need to be done to get Flink working(Other than > downloading > > > the correct binaries, which I believe I did) > > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > > hadoop27-scala_2.11.tgz > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > hadoop27-scala_2.11.tgz>) > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > I appreciate any help or guidance you can provide me in fixing my > > problems, > > > please let me know if there is anything else I can provide you. > > > > > > Thank you > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email]> > wrote: > > > > > > > Hi Andy, > > > > > > > > I'm not an AWS expert, so I'll just check on some common issues. > > > > > > > > I guess you already had a look at the Flink docs for AWS/EMR but I'll > > > post > > > > the link just be to sure [1]. > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) > did > > > you > > > > built Flink yourself or did you download the binaries? > > > > Does the Hadoop version of the Flink build match the Hadoop version > of > > > EMR > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > Best, Fabian > > > > > > > > [1] > > > > https://ci.apache.org/projects/flink/flink-docs- > > > release-1.3/setup/aws.html > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > Hi Fabian, > > > > > > > > > > Sorry, I just realized I forgot to include that part. The error > > > returned > > > > > is: > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > Lorg/apache/hadoop/conf/ > > > > > Configuration;)V > > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > > > EmrFileSystem.java:93) > > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > > initialize(HadoopFileSystem.java:328) > > > > > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem( > > > > > FileSystem.java:350) > > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem. > java:389) > > > > > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293) > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory. > java:99) > > > > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > at org.apache.flink.contrib.streaming.state. > RocksDBStateBackend. > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > I believe it has something to do with the classpath, but I am > unsure > > > why > > > > or > > > > > how to fix it. The classpath being used during the execution is: > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it seems > > the > > > > > addResource function does seem to be there. > > > > > > > > > > Thank you > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email]> > > > wrote: > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > can you describe in more detail what exactly isn't working? > > > > > > Do you see error messages in the log files or on the console? > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I am about to deploy my first Flink projects to production, > but > > I > > > am > > > > > > > running into a very big hurdle. I am unable to launch my > project > > > so > > > > it > > > > > > can > > > > > > > write to an S3 bucket. My project is running on an EMR > cluster, > > > > where > > > > > I > > > > > > > have installed Flink 1.3.2. I am using Yarn to launch the > > > > application, > > > > > > and > > > > > > > it seems to run fine unless I am trying to enable check > > > > pointing(with a > > > > > > S3 > > > > > > > target). I am looking to use RocksDB as my check-pointing > > backend. > > > > I > > > > > > have > > > > > > > asked a few places, and I am still unable to find a solution to > > > this > > > > > > > problem. Here are my steps for creating a cluster, and > launching > > > my > > > > > > > application, perhaps I am missing a step. I'd be happy to > > provide > > > > any > > > > > > > additional information if needed. > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > 2) Advanced Options > > > > > > > 3) Release = emr-5.8.0 > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > > > > > names/keys/etc) > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > > following: > > > > > > > > > > > > > > 1 wget > > > > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > > > 3 cd flink-1.3.2 > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > ~/flink-consumer.jar > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > CheckpointingMode.EXACTLY_ > > > > > > ONCE) > > > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > > getCheckpointMinPause) > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > getCheckpointTimeout) > > > > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > > bucket/location", > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hi Till,
I believe this is what you are looking for, classpath is much bigger for the task manager. I can also post the whole log file if needed: 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn.YarnTaskManagerRunner - Classpath: flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink-python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3-amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.160.jar Thank you On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <[hidden email]> wrote: > Hi Andy, > > the CliFrontend is not executed via Yarn, thus, it is not affected by > dependencies which are added due to the underlying Yarn cluster. Therefore, > it would be helpful to look at the TaskManager logs. Either you have > enabled log aggregation on your Yarn cluster, then you can obtain the logs > via `yarn logs -applicationId <application ID>` or you have to retrieve > them from the machines where they were running (either by going directly > there or via the Yarn web interface). > > Cheers, > Till > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> wrote: > > > Hi Till, > > > > That is actually the classpath used by the flink bash script(that > launches > > the jar using the java command). I changed the execute to an echo, and > > grabbed that for the CLI arguments. > > > > I believe this is the class path from the log file(although it might not > be > > the taskmanager log, is that any different from what would be in my > > flink-1.3.2/log folder?): > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend > > - Classpath: > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > > > > If that doesn't seem right, and you can point me in the right direction > as > > to where the TaskManager logs would be, I would be happy to grab the > > information your looking for. > > > > Thank you > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Andy, > > > > > > this looks to me indeed like a dependency problem. I assume that EMR or > > > something else is pulling in an incompatible version of Hadoop. > > > > > > The classpath you've posted, is this the one logged in the log files > > > (TaskManager log) or did you compile it yourself? In the latter case, > it > > > would also be helpful to get access to the TaskManager logs. > > > > > > Cheers, > > > Till > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> wrote: > > > > > > > Hi Fabian, > > > > > > > > 1) I have looked at the linked docs, and from what I can tell no > setup > > > > should really need to be done to get Flink working(Other than > > downloading > > > > the correct binaries, which I believe I did) > > > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > > > hadoop27-scala_2.11.tgz > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > hadoop27-scala_2.11.tgz>) > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > I appreciate any help or guidance you can provide me in fixing my > > > problems, > > > > please let me know if there is anything else I can provide you. > > > > > > > > Thank you > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email]> > > wrote: > > > > > > > > > Hi Andy, > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common issues. > > > > > > > > > > I guess you already had a look at the Flink docs for AWS/EMR but > I'll > > > > post > > > > > the link just be to sure [1]. > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) > > did > > > > you > > > > > built Flink yourself or did you download the binaries? > > > > > Does the Hadoop version of the Flink build match the Hadoop version > > of > > > > EMR > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > Best, Fabian > > > > > > > > > > [1] > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > release-1.3/setup/aws.html > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. The error > > > > returned > > > > > > is: > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > Lorg/apache/hadoop/conf/ > > > > > > Configuration;)V > > > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > > > > EmrFileSystem.java:93) > > > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > at org.apache.flink.core.fs.FileSystem. > getUnguardedFileSystem( > > > > > > FileSystem.java:350) > > > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem. > > java:389) > > > > > > at org.apache.flink.core.fs.Path. > getFileSystem(Path.java:293) > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory. > > java:99) > > > > > > at org.apache.flink.runtime.state.filesystem.FsStateBackend. > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > at org.apache.flink.contrib.streaming.state. > > RocksDBStateBackend. > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > I believe it has something to do with the classpath, but I am > > unsure > > > > why > > > > > or > > > > > > how to fix it. The classpath being used during the execution is: > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it > seems > > > the > > > > > > addResource function does seem to be there. > > > > > > > > > > > > Thank you > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <[hidden email] > > > > > > wrote: > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > can you describe in more detail what exactly isn't working? > > > > > > > Do you see error messages in the log files or on the console? > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to production, > > but > > > I > > > > am > > > > > > > > running into a very big hurdle. I am unable to launch my > > project > > > > so > > > > > it > > > > > > > can > > > > > > > > write to an S3 bucket. My project is running on an EMR > > cluster, > > > > > where > > > > > > I > > > > > > > > have installed Flink 1.3.2. I am using Yarn to launch the > > > > > application, > > > > > > > and > > > > > > > > it seems to run fine unless I am trying to enable check > > > > > pointing(with a > > > > > > > S3 > > > > > > > > target). I am looking to use RocksDB as my check-pointing > > > backend. > > > > > I > > > > > > > have > > > > > > > > asked a few places, and I am still unable to find a solution > to > > > > this > > > > > > > > problem. Here are my steps for creating a cluster, and > > launching > > > > my > > > > > > > > application, perhaps I am missing a step. I'd be happy to > > > provide > > > > > any > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > 2) Advanced Options > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill out > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > > > following: > > > > > > > > > > > > > > > > 1 wget > > > > > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > CheckpointingMode.EXACTLY_ > > > > > > > ONCE) > > > > > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > > > getCheckpointMinPause) > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > getCheckpointTimeout) > > > > > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > > > bucket/location", > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hi Andy,
could you check which Hadoop version this jar /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the contained hadoop Configuration class has the method Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe this jar is the culprit because it comes from a different Hadoop version. Cheers, Till On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > Hi Till, > > I believe this is what you are looking for, classpath is much bigger for > the task manager. I can also post the whole log file if needed: > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn.YarnTaskManagerRunner > - Classpath: > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop- > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/commons- > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/ > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > hadoop-yarn/hadoop-yarn-applications-distributedshell- > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > hadoop-yarn-server-applicationhistoryservice.jar: > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11. > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk-iot-1.11.160.jar > > Thank you > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Andy, > > > > the CliFrontend is not executed via Yarn, thus, it is not affected by > > dependencies which are added due to the underlying Yarn cluster. > Therefore, > > it would be helpful to look at the TaskManager logs. Either you have > > enabled log aggregation on your Yarn cluster, then you can obtain the > logs > > via `yarn logs -applicationId <application ID>` or you have to retrieve > > them from the machines where they were running (either by going directly > > there or via the Yarn web interface). > > > > Cheers, > > Till > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> wrote: > > > > > Hi Till, > > > > > > That is actually the classpath used by the flink bash script(that > > launches > > > the jar using the java command). I changed the execute to an echo, and > > > grabbed that for the CLI arguments. > > > > > > I believe this is the class path from the log file(although it might > not > > be > > > the taskmanager log, is that any different from what would be in my > > > flink-1.3.2/log folder?): > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend > > > - Classpath: > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > > > > > > If that doesn't seem right, and you can point me in the right direction > > as > > > to where the TaskManager logs would be, I would be happy to grab the > > > information your looking for. > > > > > > Thank you > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Andy, > > > > > > > > this looks to me indeed like a dependency problem. I assume that EMR > or > > > > something else is pulling in an incompatible version of Hadoop. > > > > > > > > The classpath you've posted, is this the one logged in the log files > > > > (TaskManager log) or did you compile it yourself? In the latter case, > > it > > > > would also be helpful to get access to the TaskManager logs. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> wrote: > > > > > > > > > Hi Fabian, > > > > > > > > > > 1) I have looked at the linked docs, and from what I can tell no > > setup > > > > > should really need to be done to get Flink working(Other than > > > downloading > > > > > the correct binaries, which I believe I did) > > > > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > > > > hadoop27-scala_2.11.tgz > > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > hadoop27-scala_2.11.tgz>) > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > I appreciate any help or guidance you can provide me in fixing my > > > > problems, > > > > > please let me know if there is anything else I can provide you. > > > > > > > > > > Thank you > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email]> > > > wrote: > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common issues. > > > > > > > > > > > > I guess you already had a look at the Flink docs for AWS/EMR but > > I'll > > > > > post > > > > > > the link just be to sure [1]. > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink > 1.3.1) > > > did > > > > > you > > > > > > built Flink yourself or did you download the binaries? > > > > > > Does the Hadoop version of the Flink build match the Hadoop > version > > > of > > > > > EMR > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > [1] > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. The > error > > > > > returned > > > > > > > is: > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > Configuration;)V > > > > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > > > > > EmrFileSystem.java:93) > > > > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > at org.apache.flink.core.fs.FileSystem. > > getUnguardedFileSystem( > > > > > > > FileSystem.java:350) > > > > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem. > > > java:389) > > > > > > > at org.apache.flink.core.fs.Path. > > getFileSystem(Path.java:293) > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory. > > > java:99) > > > > > > > at org.apache.flink.runtime.state.filesystem. > FsStateBackend. > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > at org.apache.flink.contrib.streaming.state. > > > RocksDBStateBackend. > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > I believe it has something to do with the classpath, but I am > > > unsure > > > > > why > > > > > > or > > > > > > > how to fix it. The classpath being used during the execution > is: > > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > > > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it > > seems > > > > the > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't working? > > > > > > > > Do you see error messages in the log files or on the console? > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > production, > > > but > > > > I > > > > > am > > > > > > > > > running into a very big hurdle. I am unable to launch my > > > project > > > > > so > > > > > > it > > > > > > > > can > > > > > > > > > write to an S3 bucket. My project is running on an EMR > > > cluster, > > > > > > where > > > > > > > I > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to launch the > > > > > > application, > > > > > > > > and > > > > > > > > > it seems to run fine unless I am trying to enable check > > > > > > pointing(with a > > > > > > > > S3 > > > > > > > > > target). I am looking to use RocksDB as my check-pointing > > > > backend. > > > > > > I > > > > > > > > have > > > > > > > > > asked a few places, and I am still unable to find a > solution > > to > > > > > this > > > > > > > > > problem. Here are my steps for creating a cluster, and > > > launching > > > > > my > > > > > > > > > application, perhaps I am missing a step. I'd be happy to > > > > provide > > > > > > any > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > 2) Advanced Options > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill > out > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > > > > following: > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > > > > getCheckpointMinPause) > > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > > getCheckpointTimeout) > > > > > > > > > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > > > > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > > > > bucket/location", > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hi Till,
Seems like everything is in line there. hadoop-common.jar -> hadoop-common-2.7.3-amzn-3.jar And when i decompiled that jar I see public void addResource(Configuration conf) in org/apache/hadoop/conf/Configuration.java I agree that an incorrect version of the jar is probably being run, is there a way to limit the classpath for the TaskManager when starting the job? Thank you On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> wrote: > Hi Andy, > > could you check which Hadoop version this jar > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the > contained hadoop Configuration class has the method > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe > this jar is the culprit because it comes from a different Hadoop version. > > Cheers, > Till > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > > > Hi Till, > > > > I believe this is what you are looking for, classpath is much bigger for > > the task manager. I can also post the whole log file if needed: > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > YarnTaskManagerRunner > > - Classpath: > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > hadoop-mapreduce/hadoop- > > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > 8.jar:/usr/lib/hadoop- > > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/commons- > > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > lib/hadoop-mapreduce/lib/ > > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > > hadoop-yarn/hadoop-yarn-applications-distributedshell- > > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > hadoop-yarn-server-applicationhistoryservice.jar: > > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > stepfunctions-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > sdk/aws-java-sdk- > > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > organizations-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > sdk/aws-java-sdk- > > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11. > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk-iot-1.11.160.jar > > > > Thank you > > > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Andy, > > > > > > the CliFrontend is not executed via Yarn, thus, it is not affected by > > > dependencies which are added due to the underlying Yarn cluster. > > Therefore, > > > it would be helpful to look at the TaskManager logs. Either you have > > > enabled log aggregation on your Yarn cluster, then you can obtain the > > logs > > > via `yarn logs -applicationId <application ID>` or you have to retrieve > > > them from the machines where they were running (either by going > directly > > > there or via the Yarn web interface). > > > > > > Cheers, > > > Till > > > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> wrote: > > > > > > > Hi Till, > > > > > > > > That is actually the classpath used by the flink bash script(that > > > launches > > > > the jar using the java command). I changed the execute to an echo, > and > > > > grabbed that for the CLI arguments. > > > > > > > > I believe this is the class path from the log file(although it might > > not > > > be > > > > the taskmanager log, is that any different from what would be in my > > > > flink-1.3.2/log folder?): > > > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend > > > > - Classpath: > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > > > > > > > > If that doesn't seem right, and you can point me in the right > direction > > > as > > > > to where the TaskManager logs would be, I would be happy to grab the > > > > information your looking for. > > > > > > > > Thank you > > > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > Hi Andy, > > > > > > > > > > this looks to me indeed like a dependency problem. I assume that > EMR > > or > > > > > something else is pulling in an incompatible version of Hadoop. > > > > > > > > > > The classpath you've posted, is this the one logged in the log > files > > > > > (TaskManager log) or did you compile it yourself? In the latter > case, > > > it > > > > > would also be helpful to get access to the TaskManager logs. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> > wrote: > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > 1) I have looked at the linked docs, and from what I can tell no > > > setup > > > > > > should really need to be done to get Flink working(Other than > > > > downloading > > > > > > the correct binaries, which I believe I did) > > > > > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > > > > > hadoop27-scala_2.11.tgz > > > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > hadoop27-scala_2.11.tgz>) > > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > > > I appreciate any help or guidance you can provide me in fixing my > > > > > problems, > > > > > > please let me know if there is anything else I can provide you. > > > > > > > > > > > > Thank you > > > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <[hidden email] > > > > > > wrote: > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common > issues. > > > > > > > > > > > > > > I guess you already had a look at the Flink docs for AWS/EMR > but > > > I'll > > > > > > post > > > > > > > the link just be to sure [1]. > > > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink > > 1.3.1) > > > > did > > > > > > you > > > > > > > built Flink yourself or did you download the binaries? > > > > > > > Does the Hadoop version of the Flink build match the Hadoop > > version > > > > of > > > > > > EMR > > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > [1] > > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. The > > error > > > > > > returned > > > > > > > > is: > > > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > > Configuration;)V > > > > > > > > at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize( > > > > > > > > EmrFileSystem.java:93) > > > > > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > > at org.apache.flink.core.fs.FileSystem. > > > getUnguardedFileSystem( > > > > > > > > FileSystem.java:350) > > > > > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem. > > > > java:389) > > > > > > > > at org.apache.flink.core.fs.Path. > > > getFileSystem(Path.java:293) > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > > FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory. > > > > java:99) > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > FsStateBackend. > > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > > at org.apache.flink.contrib.streaming.state. > > > > RocksDBStateBackend. > > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > > > I believe it has something to do with the classpath, but I am > > > > unsure > > > > > > why > > > > > > > or > > > > > > > > how to fix it. The classpath being used during the execution > > is: > > > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > > > > uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2. > > > > > > > > 17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > > jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3. > > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and it > > > seems > > > > > the > > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't working? > > > > > > > > > Do you see error messages in the log files or on the > console? > > > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > > production, > > > > but > > > > > I > > > > > > am > > > > > > > > > > running into a very big hurdle. I am unable to launch my > > > > project > > > > > > so > > > > > > > it > > > > > > > > > can > > > > > > > > > > write to an S3 bucket. My project is running on an EMR > > > > cluster, > > > > > > > where > > > > > > > > I > > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to launch > the > > > > > > > application, > > > > > > > > > and > > > > > > > > > > it seems to run fine unless I am trying to enable check > > > > > > > pointing(with a > > > > > > > > > S3 > > > > > > > > > > target). I am looking to use RocksDB as my > check-pointing > > > > > backend. > > > > > > > I > > > > > > > > > have > > > > > > > > > > asked a few places, and I am still unable to find a > > solution > > > to > > > > > > this > > > > > > > > > > problem. Here are my steps for creating a cluster, and > > > > launching > > > > > > my > > > > > > > > > > application, perhaps I am missing a step. I'd be happy > to > > > > > provide > > > > > > > any > > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > > 2) Advanced Options > > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do fill > > out > > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > > > > > following: > > > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > > http://apache.claz.org/flink/ > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz > > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > > state.backend.fs.checkpointdir: s3:/bucket/location > > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > > > > > getCheckpointMinPause) > > > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > > > getCheckpointTimeout) > > > > > > > > > > env.getCheckpointConfig. > setMaxConcurrentCheckpoints(1) > > > > > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > > > > > bucket/location", > > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hi Andy,
I believe it's because you didn't set your s3 impl correctly. Try to set your core-site.xml by following https://ci.apache.org/ projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#s3afilesystem- recommended Bowen On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: > Hi Till, > > Seems like everything is in line there. hadoop-common.jar -> > hadoop-common-2.7.3-amzn-3.jar > > And when i decompiled that jar I see public void addResource(Configuration > conf) in org/apache/hadoop/conf/Configuration.java > > I agree that an incorrect version of the jar is probably being run, is > there a way to limit the classpath for the TaskManager when starting the > job? > > Thank you > > On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Andy, > > > > could you check which Hadoop version this jar > > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the > > contained hadoop Configuration class has the method > > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. > Maybe > > this jar is the culprit because it comes from a different Hadoop version. > > > > Cheers, > > Till > > > > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > > > > > Hi Till, > > > > > > I believe this is what you are looking for, classpath is much bigger > for > > > the task manager. I can also post the whole log file if needed: > > > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > > YarnTaskManagerRunner > > > - Classpath: > > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > > > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > > > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > > > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > > > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > > > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > > > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > > > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > > > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > > > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > > > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > > > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > > > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > > > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > > > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > > > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > > > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > > > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > > > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > > > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > > > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > > > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > > > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > > > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > > > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > > > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > > > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > > > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > > > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > > > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > > > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > > > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > > > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > > > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > > > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > > > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > > > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > > > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > > > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > > > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > > > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > > > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > > > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > > > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > > hadoop-mapreduce/hadoop- > > > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > > > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > > > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > > > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > > > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > > > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > > > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > > > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > > > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > > > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > > > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > > > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > > 8.jar:/usr/lib/hadoop- > > > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > > > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > > > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > > > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > > > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > > > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > > > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > > > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > > > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > > > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > > > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > > > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > > > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > > > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > > > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > > > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > > > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > > > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > > > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > > > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > > > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > > > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > > > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > > > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > > > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > > > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > > > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > > > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > > > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > > > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > > > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > > > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > > > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > > > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > > > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > > > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > > > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > > > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > > > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > > > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > > > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > > > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > > > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > > > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > > > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > > > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > > > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > > > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-mapreduce/commons- > > > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > > > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > > > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > > > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > > > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > > > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > > > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > > > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > > > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > > > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > > > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > > > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > > > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > > > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > > > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > > > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > > lib/hadoop-mapreduce/lib/ > > > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > > > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > > > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > > > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > > > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > > > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > > > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > > > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > > > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > > > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > > > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > > > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > > > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > > > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > > > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > > > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > > > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > > > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > > > hadoop-yarn/hadoop-yarn-applications-distributedshell- > > > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > hadoop-yarn-server-applicationhistoryservice.jar: > > > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > > > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > > > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > > > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > > > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > > > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > > > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > > > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > > > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > > > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > > > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > > > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > > > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > > > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > > > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > > > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > > > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > > > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > > > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > > > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > > > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > > > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > > > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > > > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > > > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > > > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > > > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > > > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > > > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > > > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > > > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > > > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > > > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > > > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > > > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > > > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > > > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > > > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > > > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > > > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > > > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > > > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > > > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > > > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > > > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > > > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > > > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > > > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > > > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > aws-java-sdk- > > > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > > > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > > > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > stepfunctions-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > > sdk/aws-java-sdk- > > > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > elastictranscoder-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > organizations-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > > sdk/aws-java-sdk- > > > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > > > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11. > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > > > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > > > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > > > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > > > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk-iot-1.11.160.jar > > > > > > Thank you > > > > > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Andy, > > > > > > > > the CliFrontend is not executed via Yarn, thus, it is not affected by > > > > dependencies which are added due to the underlying Yarn cluster. > > > Therefore, > > > > it would be helpful to look at the TaskManager logs. Either you have > > > > enabled log aggregation on your Yarn cluster, then you can obtain the > > > logs > > > > via `yarn logs -applicationId <application ID>` or you have to > retrieve > > > > them from the machines where they were running (either by going > > directly > > > > there or via the Yarn web interface). > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> wrote: > > > > > > > > > Hi Till, > > > > > > > > > > That is actually the classpath used by the flink bash script(that > > > > launches > > > > > the jar using the java command). I changed the execute to an echo, > > and > > > > > grabbed that for the CLI arguments. > > > > > > > > > > I believe this is the class path from the log file(although it > might > > > not > > > > be > > > > > the taskmanager log, is that any different from what would be in my > > > > > flink-1.3.2/log folder?): > > > > > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client.CliFrontend > > > > > - Classpath: > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ > hadoop/conf: > > > > > > > > > > If that doesn't seem right, and you can point me in the right > > direction > > > > as > > > > > to where the TaskManager logs would be, I would be happy to grab > the > > > > > information your looking for. > > > > > > > > > > Thank you > > > > > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < > [hidden email]> > > > > > wrote: > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > this looks to me indeed like a dependency problem. I assume that > > EMR > > > or > > > > > > something else is pulling in an incompatible version of Hadoop. > > > > > > > > > > > > The classpath you've posted, is this the one logged in the log > > files > > > > > > (TaskManager log) or did you compile it yourself? In the latter > > case, > > > > it > > > > > > would also be helpful to get access to the TaskManager logs. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> > > wrote: > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > 1) I have looked at the linked docs, and from what I can tell > no > > > > setup > > > > > > > should really need to be done to get Flink working(Other than > > > > > downloading > > > > > > > the correct binaries, which I believe I did) > > > > > > > 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > hadoop27-scala_2.11.tgz>) > > > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > > > > > I appreciate any help or guidance you can provide me in fixing > my > > > > > > problems, > > > > > > > please let me know if there is anything else I can provide you. > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common > > issues. > > > > > > > > > > > > > > > > I guess you already had a look at the Flink docs for AWS/EMR > > but > > > > I'll > > > > > > > post > > > > > > > > the link just be to sure [1]. > > > > > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink > > > 1.3.1) > > > > > did > > > > > > > you > > > > > > > > built Flink yourself or did you download the binaries? > > > > > > > > Does the Hadoop version of the Flink build match the Hadoop > > > version > > > > > of > > > > > > > EMR > > > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > [1] > > > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. The > > > error > > > > > > > returned > > > > > > > > > is: > > > > > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > > > Configuration;)V > > > > > > > > > at com.amazon.ws.emr.hadoop.fs. > EmrFileSystem.initialize( > > > > > > > > > EmrFileSystem.java:93) > > > > > > > > > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem. > > > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > > > at org.apache.flink.core.fs.FileSystem. > > > > getUnguardedFileSystem( > > > > > > > > > FileSystem.java:350) > > > > > > > > > at org.apache.flink.core.fs.FileSystem.get(FileSystem. > > > > > java:389) > > > > > > > > > at org.apache.flink.core.fs.Path. > > > > getFileSystem(Path.java:293) > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > > > FsCheckpointStreamFactory.<init>( > FsCheckpointStreamFactory. > > > > > java:99) > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > FsStateBackend. > > > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > > > at org.apache.flink.contrib.streaming.state. > > > > > RocksDBStateBackend. > > > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > > > > > I believe it has something to do with the classpath, but I > am > > > > > unsure > > > > > > > why > > > > > > > > or > > > > > > > > > how to fix it. The classpath being used during the > execution > > > is: > > > > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > > > > > uber-1.3.2.jar:/home/hadoop/ > flink-1.3.2/lib/log4j-1.2. > > > > > > > > > 17.jar:/home/hadoop/flink-1. > 3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > > > jar:/home/hadoop/flink-1. > 3.2/lib/flink-dist_2.11-1.3. > > > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and > it > > > > seems > > > > > > the > > > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't > working? > > > > > > > > > > Do you see error messages in the log files or on the > > console? > > > > > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > > > production, > > > > > but > > > > > > I > > > > > > > am > > > > > > > > > > > running into a very big hurdle. I am unable to launch > my > > > > > project > > > > > > > so > > > > > > > > it > > > > > > > > > > can > > > > > > > > > > > write to an S3 bucket. My project is running on an EMR > > > > > cluster, > > > > > > > > where > > > > > > > > > I > > > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to launch > > the > > > > > > > > application, > > > > > > > > > > and > > > > > > > > > > > it seems to run fine unless I am trying to enable check > > > > > > > > pointing(with a > > > > > > > > > > S3 > > > > > > > > > > > target). I am looking to use RocksDB as my > > check-pointing > > > > > > backend. > > > > > > > > I > > > > > > > > > > have > > > > > > > > > > > asked a few places, and I am still unable to find a > > > solution > > > > to > > > > > > > this > > > > > > > > > > > problem. Here are my steps for creating a cluster, and > > > > > launching > > > > > > > my > > > > > > > > > > > application, perhaps I am missing a step. I'd be happy > > to > > > > > > provide > > > > > > > > any > > > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > > > 2) Advanced Options > > > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do > fill > > > out > > > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do the > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > > > http://apache.claz.org/flink/ > > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27- > scala_2.11.tgz > > > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > > > state.backend.fs.checkpointdir: > s3:/bucket/location > > > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig.enableExternalizedCheckpoints( > > > > > > > > > > > ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig.setMinPauseBetweenCheckpoints( > > > > > > > > > > > getCheckpointMinPause) > > > > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > > > > getCheckpointTimeout) > > > > > > > > > > > env.getCheckpointConfig. > > setMaxConcurrentCheckpoints(1) > > > > > > > > > > > env.setStateBackend(new RocksDBStateBackend("s3:// > > > > > > > > > bucket/location", > > > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Attached my side project verified working to deploy jobmanager and
taskmanager as stateless service(non yarn/mesos), configuration here https://github.com/chenqin/flink-jar/tree/master/config/hadoop more detail here https://github.com/chenqin/flink-jar/blob/master/src/main/java/FlinkBootstrap.java#L49 On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <[hidden email]> wrote: > Hi Andy, > > I believe it's because you didn't set your s3 impl correctly. Try to set > your core-site.xml by following https://ci.apache.org/ > projects/flink/flink-docs-release-1.4/ops/deployment/ > aws.html#s3afilesystem- > recommended > > Bowen > > On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: > > > Hi Till, > > > > Seems like everything is in line there. hadoop-common.jar -> > > hadoop-common-2.7.3-amzn-3.jar > > > > And when i decompiled that jar I see public void > addResource(Configuration > > conf) in org/apache/hadoop/conf/Configuration.java > > > > I agree that an incorrect version of the jar is probably being run, is > > there a way to limit the classpath for the TaskManager when starting the > > job? > > > > Thank you > > > > On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Andy, > > > > > > could you check which Hadoop version this jar > > > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the > > > contained hadoop Configuration class has the method > > > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. > > Maybe > > > this jar is the culprit because it comes from a different Hadoop > version. > > > > > > Cheers, > > > Till > > > > > > > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > > > > > > > Hi Till, > > > > > > > > I believe this is what you are looking for, classpath is much bigger > > for > > > > the task manager. I can also post the whole log file if needed: > > > > > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > > > YarnTaskManagerRunner > > > > - Classpath: > > > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > > > > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > > > > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > > > > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > > > > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > > > > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > > > > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > > > > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > > > > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > > > > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > > > > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > > > > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > > > > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > > > > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > > > > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > > > > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > > > > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > > > > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > > > > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > > > > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > > > > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > > > > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > > > > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > > > > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > > > > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > > > > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > > > > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > > > > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > > > > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > > > > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > > > > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > > > > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > > > > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > > > > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > > > > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > > > > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > > > > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > > > > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > > > > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > > > > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > > > > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > > > hadoop-mapreduce/hadoop- > > > > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > > > > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > > > > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > > > > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > > > > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > > > > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > > > > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > > > > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > > > > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > > > > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > > > > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > > > > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > > > 8.jar:/usr/lib/hadoop- > > > > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > > > > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > > > > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > > > > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > > > > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > > > > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > > > > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > > > > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > > > > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > > > > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > > > > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > > > > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > > > > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > > > > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > > > > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > > > > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > > > > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > > > > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > > > > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > > > > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > > > > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > > > > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > > > > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > > > > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > > > > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > > > > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > > > > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > > > > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > > > > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > > > > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > > > > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > > > > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > > > > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > > > > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > > > > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > > > > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > > > > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > > > > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > > > > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > > > > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > > > > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > > > > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > > > > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > > > > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > > > > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- > mapreduce/commons- > > > > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > > > > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > > > > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > > > > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > > > > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > > > > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > > > > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > > > > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > > > > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > > > > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > > > > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > > > > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > > > > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > > > > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > > > > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > > > > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > > > lib/hadoop-mapreduce/lib/ > > > > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > > > > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > > > > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > > > > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > > > > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > > > > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > > > > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > > > > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > > > > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > > > > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > > > > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > > > > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > > > > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > > > > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- > yarn-server- > > > > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > > > > hadoop-yarn/hadoop-yarn-applications-distributedshell- > > > > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > hadoop-yarn-server-applicationhistoryservice.jar: > > > > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > > > > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > > > > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > > > > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > > > > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > > > > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > > > > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > > > > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > > > > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > > > > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > > > > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > > > > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > > > > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > > > > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > > > > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > > > > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > > > > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > > > > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > > > > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > > > > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > > > > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > > > > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > > > > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > > > > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > > > > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > > > > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > > > > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > > > > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > > > > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > > > > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > > > > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > > > > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > > > > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > > > > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > > > > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > > > > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > > > > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > > > > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > > > > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > > > > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > > > > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > > > > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > > > > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > > > > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > > > > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > > > > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > aws-java-sdk- > > > > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > > > > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > > > > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > stepfunctions-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > > > sdk/aws-java-sdk- > > > > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > elastictranscoder-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > organizations-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > > > sdk/aws-java-sdk- > > > > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > > > > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11. > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > > > > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > > > > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > > > > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk-iot-1.11.160.jar > > > > > > > > Thank you > > > > > > > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > Hi Andy, > > > > > > > > > > the CliFrontend is not executed via Yarn, thus, it is not affected > by > > > > > dependencies which are added due to the underlying Yarn cluster. > > > > Therefore, > > > > > it would be helpful to look at the TaskManager logs. Either you > have > > > > > enabled log aggregation on your Yarn cluster, then you can obtain > the > > > > logs > > > > > via `yarn logs -applicationId <application ID>` or you have to > > retrieve > > > > > them from the machines where they were running (either by going > > > directly > > > > > there or via the Yarn web interface). > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > That is actually the classpath used by the flink bash script(that > > > > > launches > > > > > > the jar using the java command). I changed the execute to an > echo, > > > and > > > > > > grabbed that for the CLI arguments. > > > > > > > > > > > > I believe this is the class path from the log file(although it > > might > > > > not > > > > > be > > > > > > the taskmanager log, is that any different from what would be in > my > > > > > > flink-1.3.2/log folder?): > > > > > > > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client. > CliFrontend > > > > > > - Classpath: > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ > > hadoop/conf: > > > > > > > > > > > > If that doesn't seem right, and you can point me in the right > > > direction > > > > > as > > > > > > to where the TaskManager logs would be, I would be happy to grab > > the > > > > > > information your looking for. > > > > > > > > > > > > Thank you > > > > > > > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > this looks to me indeed like a dependency problem. I assume > that > > > EMR > > > > or > > > > > > > something else is pulling in an incompatible version of Hadoop. > > > > > > > > > > > > > > The classpath you've posted, is this the one logged in the log > > > files > > > > > > > (TaskManager log) or did you compile it yourself? In the latter > > > case, > > > > > it > > > > > > > would also be helpful to get access to the TaskManager logs. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> > > > wrote: > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > 1) I have looked at the linked docs, and from what I can tell > > no > > > > > setup > > > > > > > > should really need to be done to get Flink working(Other than > > > > > > downloading > > > > > > > > the correct binaries, which I believe I did) > > > > > > > > 2) I have downloaded the Flink 1.3.2 > binaries(flink-1.3.2-bin- > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > > hadoop27-scala_2.11.tgz>) > > > > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > > > > > > > I appreciate any help or guidance you can provide me in > fixing > > my > > > > > > > problems, > > > > > > > > please let me know if there is anything else I can provide > you. > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common > > > issues. > > > > > > > > > > > > > > > > > > I guess you already had a look at the Flink docs for > AWS/EMR > > > but > > > > > I'll > > > > > > > > post > > > > > > > > > the link just be to sure [1]. > > > > > > > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink > > > > 1.3.1) > > > > > > did > > > > > > > > you > > > > > > > > > built Flink yourself or did you download the binaries? > > > > > > > > > Does the Hadoop version of the Flink build match the Hadoop > > > > version > > > > > > of > > > > > > > > EMR > > > > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. > The > > > > error > > > > > > > > returned > > > > > > > > > > is: > > > > > > > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > > > > Configuration;)V > > > > > > > > > > at com.amazon.ws.emr.hadoop.fs. > > EmrFileSystem.initialize( > > > > > > > > > > EmrFileSystem.java:93) > > > > > > > > > > at org.apache.flink.runtime.fs. > hdfs.HadoopFileSystem. > > > > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > > > > at org.apache.flink.core.fs.FileSystem. > > > > > getUnguardedFileSystem( > > > > > > > > > > FileSystem.java:350) > > > > > > > > > > at org.apache.flink.core.fs. > FileSystem.get(FileSystem. > > > > > > java:389) > > > > > > > > > > at org.apache.flink.core.fs.Path. > > > > > getFileSystem(Path.java:293) > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > > > > FsCheckpointStreamFactory.<init>( > > FsCheckpointStreamFactory. > > > > > > java:99) > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > FsStateBackend. > > > > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > > > > at org.apache.flink.contrib.streaming.state. > > > > > > RocksDBStateBackend. > > > > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > > > > > > > I believe it has something to do with the classpath, but > I > > am > > > > > > unsure > > > > > > > > why > > > > > > > > > or > > > > > > > > > > how to fix it. The classpath being used during the > > execution > > > > is: > > > > > > > > > > /home/hadoop/flink-1.3.2/lib/ > flink-python_2.11-1.3.2.jar:/ > > > > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2- > > > > > > > > > > uber-1.3.2.jar:/home/hadoop/ > > flink-1.3.2/lib/log4j-1.2. > > > > > > > > > > 17.jar:/home/hadoop/flink-1. > > 3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > > > > jar:/home/hadoop/flink-1. > > 3.2/lib/flink-dist_2.11-1.3. > > > > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar and > > it > > > > > seems > > > > > > > the > > > > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't > > working? > > > > > > > > > > > Do you see error messages in the log files or on the > > > console? > > > > > > > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. <[hidden email] > >: > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > > > > production, > > > > > > but > > > > > > > I > > > > > > > > am > > > > > > > > > > > > running into a very big hurdle. I am unable to > launch > > my > > > > > > project > > > > > > > > so > > > > > > > > > it > > > > > > > > > > > can > > > > > > > > > > > > write to an S3 bucket. My project is running on an > EMR > > > > > > cluster, > > > > > > > > > where > > > > > > > > > > I > > > > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to > launch > > > the > > > > > > > > > application, > > > > > > > > > > > and > > > > > > > > > > > > it seems to run fine unless I am trying to enable > check > > > > > > > > > pointing(with a > > > > > > > > > > > S3 > > > > > > > > > > > > target). I am looking to use RocksDB as my > > > check-pointing > > > > > > > backend. > > > > > > > > > I > > > > > > > > > > > have > > > > > > > > > > > > asked a few places, and I am still unable to find a > > > > solution > > > > > to > > > > > > > > this > > > > > > > > > > > > problem. Here are my steps for creating a cluster, > and > > > > > > launching > > > > > > > > my > > > > > > > > > > > > application, perhaps I am missing a step. I'd be > happy > > > to > > > > > > > provide > > > > > > > > > any > > > > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > > > > 2) Advanced Options > > > > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I do > > fill > > > > out > > > > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do > the > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > > > > http://apache.claz.org/flink/ > > > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27- > > scala_2.11.tgz > > > > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > > > > state.backend.fs.checkpointdir: > > s3:/bucket/location > > > > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > enableExternalizedCheckpoints( > > > > > > > > > > > > ExternalizedCheckpointCleanup. > RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > setMinPauseBetweenCheckpoints( > > > > > > > > > > > > getCheckpointMinPause) > > > > > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > > > > > getCheckpointTimeout) > > > > > > > > > > > > env.getCheckpointConfig. > > > setMaxConcurrentCheckpoints(1) > > > > > > > > > > > > env.setStateBackend(new > RocksDBStateBackend("s3:// > > > > > > > > > > bucket/location", > > > > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hi!
It looks like multiple Hadoop versions are in the classpath. Flink's hadoop jar and the EMR Hadoop jars. I would simply drop Flink's own Hadoop dependency and only use the EMR Hadoop jars. Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the setup is such that the Hadoop lib environment variable is set. Then it should not have conflicts any more. On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <[hidden email]> wrote: > Attached my side project verified working to deploy jobmanager and > taskmanager as stateless service(non yarn/mesos), configuration here > > https://github.com/chenqin/flink-jar/tree/master/config/hadoop > > more detail here > https://github.com/chenqin/flink-jar/blob/master/src/ > main/java/FlinkBootstrap.java#L49 > > On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <[hidden email]> wrote: > > > Hi Andy, > > > > I believe it's because you didn't set your s3 impl correctly. Try to set > > your core-site.xml by following https://ci.apache.org/ > > projects/flink/flink-docs-release-1.4/ops/deployment/ > > aws.html#s3afilesystem- > > recommended > > > > Bowen > > > > On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: > > > > > Hi Till, > > > > > > Seems like everything is in line there. hadoop-common.jar -> > > > hadoop-common-2.7.3-amzn-3.jar > > > > > > And when i decompiled that jar I see public void > > addResource(Configuration > > > conf) in org/apache/hadoop/conf/Configuration.java > > > > > > I agree that an incorrect version of the jar is probably being run, is > > > there a way to limit the classpath for the TaskManager when starting > the > > > job? > > > > > > Thank you > > > > > > On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Andy, > > > > > > > > could you check which Hadoop version this jar > > > > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether > the > > > > contained hadoop Configuration class has the method > > > > Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. > > > Maybe > > > > this jar is the culprit because it comes from a different Hadoop > > version. > > > > > > > > Cheers, > > > > Till > > > > > > > > > > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > > > > > > > > > Hi Till, > > > > > > > > > > I believe this is what you are looking for, classpath is much > bigger > > > for > > > > > the task manager. I can also post the whole log file if needed: > > > > > > > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > > > > YarnTaskManagerRunner > > > > > - Classpath: > > > > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > > > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > > > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > > > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > > > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > > > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > > > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > > > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > > > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > > > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > > > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > > > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > > > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > > > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > > > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > > > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > > > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > > > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > > > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > > > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > > > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > > > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > > > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > > > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > > > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > > > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > > > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > > > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > > > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > > > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > > > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > > > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > > > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > > > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > > > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > > > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > > > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > > > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > > > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > > > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > > > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > > > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > > > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > > > > > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > > > > > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > > > > > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > > > > > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > > > > > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > > > > > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > > > > > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > > > > > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > > > > > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > > > > > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > > > > > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > > > > > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > > > > > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > > > > > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > > > > > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > > > > > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > > > > > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > > > > > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > > > > > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > > > > > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > > > > > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > > > > > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > > > > > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > > > > > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > > > > > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > > > > > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > > > > > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > > > > > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > > > > > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > > > > > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > > > > > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > > > > > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > > > > > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > > > > > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > > > > > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > > > > > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > > > > hadoop-mapreduce/hadoop- > > > > > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > > > > > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > > > > > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > > > > > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > > > > > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > > > > > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > > > > 8.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > > > > > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > > > > > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > > > > > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > > > > > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > > > > > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > > > > > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > > > > > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > > > > > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > > > > > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > > > > > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > > > > > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > > > > > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > > > > > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > > > > > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > > > > > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > > > > > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > > > > > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > > > > > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > > > > > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > > > > > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > > > > > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > > > > > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > > > > > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > > > > > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > > > > > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > > > > > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > > > > > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > > > > > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > > > > > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > > > > > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > > > > > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > > > > > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > > > > > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > > > > > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > > > > > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > > > > > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- > > mapreduce/commons- > > > > > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > > > > > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > > > > > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > > > > > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > > > > > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > > > > > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > > > > > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > > > > > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > > > > > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > > > > > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > > > > > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > > > > > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > > > > lib/hadoop-mapreduce/lib/ > > > > > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > > > > > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > > > > > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > > > > > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > > > > > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > > > > > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > > > > > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > > > > > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > > > > > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- > > yarn-server- > > > > > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > > > > > hadoop-yarn/hadoop-yarn-applications-distributedshell- > > > > > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- > yarn-server- > > > > > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-server-applicationhistoryservice.jar: > > > > > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > > > > > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > > > > > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > > > > > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > > > > > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > > > > > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > > > > > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > > > > > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > > > > > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > > > > > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > > > > > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > > > > > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > > > > > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > > > > > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > > > > > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > > > > > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > > > > > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > > > > > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > > > > > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > > > > > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > > > > > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > > > > > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > > > > > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > > > > > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > > > > > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > > > > > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > > > > > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > > > > > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > > > > > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > > > > > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > > > > > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > > > > > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > > > > > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > > > > > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > > > > > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > > > > > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > > > > > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > > > > > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > > > > > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > > > > > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > > > > > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > > > > > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > > > > > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > > > > > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > > > > > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > aws-java-sdk- > > > > > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > > > > > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > > > > > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > stepfunctions-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > > > > sdk/aws-java-sdk- > > > > > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > elastictranscoder-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > organizations-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > > > > sdk/aws-java-sdk- > > > > > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > > > > > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11. > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > > > > > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > > > > > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > > > > > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > aws-java-sdk-iot-1.11.160.jar > > > > > > > > > > Thank you > > > > > > > > > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < > [hidden email]> > > > > > wrote: > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > the CliFrontend is not executed via Yarn, thus, it is not > affected > > by > > > > > > dependencies which are added due to the underlying Yarn cluster. > > > > > Therefore, > > > > > > it would be helpful to look at the TaskManager logs. Either you > > have > > > > > > enabled log aggregation on your Yarn cluster, then you can obtain > > the > > > > > logs > > > > > > via `yarn logs -applicationId <application ID>` or you have to > > > retrieve > > > > > > them from the machines where they were running (either by going > > > > directly > > > > > > there or via the Yarn web interface). > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> > wrote: > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > That is actually the classpath used by the flink bash > script(that > > > > > > launches > > > > > > > the jar using the java command). I changed the execute to an > > echo, > > > > and > > > > > > > grabbed that for the CLI arguments. > > > > > > > > > > > > > > I believe this is the class path from the log file(although it > > > might > > > > > not > > > > > > be > > > > > > > the taskmanager log, is that any different from what would be > in > > my > > > > > > > flink-1.3.2/log folder?): > > > > > > > > > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client. > > CliFrontend > > > > > > > - Classpath: > > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > > > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > > > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > > > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ > > > hadoop/conf: > > > > > > > > > > > > > > If that doesn't seem right, and you can point me in the right > > > > direction > > > > > > as > > > > > > > to where the TaskManager logs would be, I would be happy to > grab > > > the > > > > > > > information your looking for. > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > this looks to me indeed like a dependency problem. I assume > > that > > > > EMR > > > > > or > > > > > > > > something else is pulling in an incompatible version of > Hadoop. > > > > > > > > > > > > > > > > The classpath you've posted, is this the one logged in the > log > > > > files > > > > > > > > (TaskManager log) or did you compile it yourself? In the > latter > > > > case, > > > > > > it > > > > > > > > would also be helpful to get access to the TaskManager logs. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <[hidden email]> > > > > wrote: > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > 1) I have looked at the linked docs, and from what I can > tell > > > no > > > > > > setup > > > > > > > > > should really need to be done to get Flink working(Other > than > > > > > > > downloading > > > > > > > > > the correct binaries, which I believe I did) > > > > > > > > > 2) I have downloaded the Flink 1.3.2 > > binaries(flink-1.3.2-bin- > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > <http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > hadoop27-scala_2.11.tgz>) > > > > > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > > > > > > > > > I appreciate any help or guidance you can provide me in > > fixing > > > my > > > > > > > > problems, > > > > > > > > > please let me know if there is anything else I can provide > > you. > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some common > > > > issues. > > > > > > > > > > > > > > > > > > > > I guess you already had a look at the Flink docs for > > AWS/EMR > > > > but > > > > > > I'll > > > > > > > > > post > > > > > > > > > > the link just be to sure [1]. > > > > > > > > > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with > Flink > > > > > 1.3.1) > > > > > > > did > > > > > > > > > you > > > > > > > > > > built Flink yourself or did you download the binaries? > > > > > > > > > > Does the Hadoop version of the Flink build match the > Hadoop > > > > > version > > > > > > > of > > > > > > > > > EMR > > > > > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email]>: > > > > > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. > > The > > > > > error > > > > > > > > > returned > > > > > > > > > > > is: > > > > > > > > > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > > > > > Configuration;)V > > > > > > > > > > > at com.amazon.ws.emr.hadoop.fs. > > > EmrFileSystem.initialize( > > > > > > > > > > > EmrFileSystem.java:93) > > > > > > > > > > > at org.apache.flink.runtime.fs. > > hdfs.HadoopFileSystem. > > > > > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > > > > > at org.apache.flink.core.fs.FileSystem. > > > > > > getUnguardedFileSystem( > > > > > > > > > > > FileSystem.java:350) > > > > > > > > > > > at org.apache.flink.core.fs. > > FileSystem.get(FileSystem. > > > > > > > java:389) > > > > > > > > > > > at org.apache.flink.core.fs.Path. > > > > > > getFileSystem(Path.java:293) > > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > > > > > FsCheckpointStreamFactory.<init>( > > > FsCheckpointStreamFactory. > > > > > > > java:99) > > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > FsStateBackend. > > > > > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > > > > > at org.apache.flink.contrib.streaming.state. > > > > > > > RocksDBStateBackend. > > > > > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > > > > > > > > > I believe it has something to do with the classpath, > but > > I > > > am > > > > > > > unsure > > > > > > > > > why > > > > > > > > > > or > > > > > > > > > > > how to fix it. The classpath being used during the > > > execution > > > > > is: > > > > > > > > > > > /home/hadoop/flink-1.3.2/lib/ > > flink-python_2.11-1.3.2.jar:/ > > > > > > > > > > > home/hadoop/flink-1.3.2/ > lib/flink-shaded-hadoop2- > > > > > > > > > > > uber-1.3.2.jar:/home/hadoop/ > > > flink-1.3.2/lib/log4j-1.2. > > > > > > > > > > > 17.jar:/home/hadoop/flink-1. > > > 3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > > > > > jar:/home/hadoop/flink-1. > > > 3.2/lib/flink-dist_2.11-1.3. > > > > > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar > and > > > it > > > > > > seems > > > > > > > > the > > > > > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't > > > working? > > > > > > > > > > > > Do you see error messages in the log files or on the > > > > console? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. < > [hidden email] > > >: > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > > > > > production, > > > > > > > but > > > > > > > > I > > > > > > > > > am > > > > > > > > > > > > > running into a very big hurdle. I am unable to > > launch > > > my > > > > > > > project > > > > > > > > > so > > > > > > > > > > it > > > > > > > > > > > > can > > > > > > > > > > > > > write to an S3 bucket. My project is running on an > > EMR > > > > > > > cluster, > > > > > > > > > > where > > > > > > > > > > > I > > > > > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to > > launch > > > > the > > > > > > > > > > application, > > > > > > > > > > > > and > > > > > > > > > > > > > it seems to run fine unless I am trying to enable > > check > > > > > > > > > > pointing(with a > > > > > > > > > > > > S3 > > > > > > > > > > > > > target). I am looking to use RocksDB as my > > > > check-pointing > > > > > > > > backend. > > > > > > > > > > I > > > > > > > > > > > > have > > > > > > > > > > > > > asked a few places, and I am still unable to find a > > > > > solution > > > > > > to > > > > > > > > > this > > > > > > > > > > > > > problem. Here are my steps for creating a cluster, > > and > > > > > > > launching > > > > > > > > > my > > > > > > > > > > > > > application, perhaps I am missing a step. I'd be > > happy > > > > to > > > > > > > > provide > > > > > > > > > > any > > > > > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > > > > > 2) Advanced Options > > > > > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I > do > > > fill > > > > > out > > > > > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and do > > the > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > > > > > http://apache.claz.org/flink/ > > > > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27- > > > scala_2.11.tgz > > > > > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d > > > > > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > > > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following fields: > > > > > > > > > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > > > > > state.backend.fs.checkpointdir: > > > s3:/bucket/location > > > > > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > > enableExternalizedCheckpoints( > > > > > > > > > > > > > ExternalizedCheckpointCleanup. > > RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > > setMinPauseBetweenCheckpoints( > > > > > > > > > > > > > getCheckpointMinPause) > > > > > > > > > > > > > env.getCheckpointConfig.setCheckpointTimeout( > > > > > > > > > > getCheckpointTimeout) > > > > > > > > > > > > > env.getCheckpointConfig. > > > > setMaxConcurrentCheckpoints(1) > > > > > > > > > > > > > env.setStateBackend(new > > RocksDBStateBackend("s3:// > > > > > > > > > > > bucket/location", > > > > > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hello,
Bowen: Unless I am missing something, it says there needs to be no setup on EMR, Each topic says: "You don’t have to configure this manually if you are running Flink on EMR." S3 access from CLI works fine on my clusters. Chen: Thank you for this, I will look into this if I am unable to get this running on YARN successfully. Stephan: Removing the said library causes the flink (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs this to work. I tried explicitly setting the classpath for the java call as well to point to the hadoop library jars. This is the original java command that I was trying to run: java -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-ip-172-31-19-27.log -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/log4j-cli.properties -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/conf/logback.xml -classpath /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 /home/hadoop/flink-consumer.jar This is what I changed it too(removing the shadded-hadoop2-uber jar and adding in the hadoop folder): java -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-ip-172-31-19-27.log -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/log4j-cli.properties -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/conf/logback.xml -classpath /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 /home/hadoop/flink-consumer.jar The later throws the following error: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/VersionInfo at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:283) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1124) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.VersionInfo at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 2 more Simply removing the .jar from the folder causes the same error. Thank you On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <[hidden email]> wrote: > Hi! > > It looks like multiple Hadoop versions are in the classpath. Flink's hadoop > jar and the EMR Hadoop jars. > I would simply drop Flink's own Hadoop dependency and only use the EMR > Hadoop jars. > > Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the > setup is such that the Hadoop lib environment variable is set. Then it > should not have conflicts any more. > > > > On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <[hidden email]> wrote: > > > Attached my side project verified working to deploy jobmanager and > > taskmanager as stateless service(non yarn/mesos), configuration here > > > > https://github.com/chenqin/flink-jar/tree/master/config/hadoop > > > > more detail here > > https://github.com/chenqin/flink-jar/blob/master/src/ > > main/java/FlinkBootstrap.java#L49 > > > > On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <[hidden email]> > wrote: > > > > > Hi Andy, > > > > > > I believe it's because you didn't set your s3 impl correctly. Try to > set > > > your core-site.xml by following https://ci.apache.org/ > > > projects/flink/flink-docs-release-1.4/ops/deployment/ > > > aws.html#s3afilesystem- > > > recommended > > > > > > Bowen > > > > > > On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: > > > > > > > Hi Till, > > > > > > > > Seems like everything is in line there. hadoop-common.jar -> > > > > hadoop-common-2.7.3-amzn-3.jar > > > > > > > > And when i decompiled that jar I see public void > > > addResource(Configuration > > > > conf) in org/apache/hadoop/conf/Configuration.java > > > > > > > > I agree that an incorrect version of the jar is probably being run, > is > > > > there a way to limit the classpath for the TaskManager when starting > > the > > > > job? > > > > > > > > Thank you > > > > > > > > On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > Hi Andy, > > > > > > > > > > could you check which Hadoop version this jar > > > > > /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether > > the > > > > > contained hadoop Configuration class has the method > > > > > Configuration.addResource(Lorg/apache/hadoop/conf/ > Configuration;)V. > > > > Maybe > > > > > this jar is the culprit because it comes from a different Hadoop > > > version. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > > > > > > On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > I believe this is what you are looking for, classpath is much > > bigger > > > > for > > > > > > the task manager. I can also post the whole log file if needed: > > > > > > > > > > > > 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > > > > > YarnTaskManagerRunner > > > > > > - Classpath: > > > > > > flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > > > > > > python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > > > > > > 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > > > > > > log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > > > > > > etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > > > > > > amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > > > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > > > > > > hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > > nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > > > > > > streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > > > > > > hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > > > > > > usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > > > > > > 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > > > > > > hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > > > > > > hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > > > > > > 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > > > > > > hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > > > > > > jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > > > > > > hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > > datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > > > > > > archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > > > > > > hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > > > > > > amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > > > > > > 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > > > > > > jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > > > > > > lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > > > > > > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > > > > > > 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > > > > > > jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > > > > > > lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > > > > > > 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > > > > > > usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > > > > > > gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > > > > > > 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > > > > > > lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > > > > > > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > > > > > > httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > > > > > > 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > > > > > > lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > > > > > > commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > > > > > > usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > > > > > > xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > > > > > > jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > > > > > > commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > > > > > > jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > > > > > > usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > > > > > > jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > > > > > > Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > > > > > > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > > > > > > api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > > > > > > 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > > > > > > /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > > > > > > jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > > > > > > cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > > > > > > lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > > > > > > apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > > > > > > commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > > > > > > 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > > > > > > /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > > > > > > lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > > > > > > 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > > > > > > usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > > > > > > lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > > > > > > jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > > > > > > hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > > > > > > curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > > > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > > > > > > 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > > > > > > usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > > > > > > hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > > > > > > amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > > > > > > incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > > > > > > 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > > > > > > jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > > > > > > lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > > > > > > lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > > > > > > commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > > > > > > jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > > > > > > usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > > > > > > hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > > > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > > > > > > 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > > > > > > hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > > > lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > > > > > > hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > > > hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > > > > > > lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > > > > > > api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > > > > > > /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > > > > > > hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > > > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > > > > > > logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > > > > > > jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > > > > > > hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > > > > > > mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > > > > > > hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > > > > > hadoop-mapreduce/hadoop- > > > > > > mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > > > > > > httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > > > mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > > > > > > lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > > > > > > mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > > commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > > jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > > > > > > mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > > > > > > jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > > > > > > lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > > > > > > usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > > > > > > hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > > > > > > 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > > > > > 8.jar:/usr/lib/hadoop- > > > > > > mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > > > > > > 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > > > > > > jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > > > > > > hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > > > > > > hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > > > > > > hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > > > > > > mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > > > > > > commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > > > > > > net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > > > > > > usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > > > > > > hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > > > > > > xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > > > > > > 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > > > > > > lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > > > > > > mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > > > > > > mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > > > > > > mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > > > > > > hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > > > > > > hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > > > > > > mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > > > > > > lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > > > mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > > > > > > mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > > mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > > > > > > 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > > > > > > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > > > > > > 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > > > > > > 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > > > > > > client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > > > > > > jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > > > > > > lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > > > common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > > > > > > 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > > > > > > usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > > > > > > jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > > > > > > usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > > > > > > mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > > > > > > mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > > > > > > hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > > > > > > hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > > > > > > lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > > > > > > mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > > mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > > > > > > archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > > > > > > 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > > > > > > jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > > > > > > lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > > > > > > mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > > > > > > jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > > > > > > kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > > > > > > jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > > > > > > /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > > > > > > /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > > > > > > usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > > > > > > lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > > > > > > lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- > > > mapreduce/commons- > > > > > > configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > > > > > > 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > > > > > > usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > > > > > > usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > > > > > > lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > > > > > > hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > > hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > > > > > > zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > > > > > > 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > > > > > > /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > > > > > > usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > > > > > > lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > > > > > > lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > > > > > > lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > > > > > > hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > > > > > > hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > > > > > > hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > > > > > lib/hadoop-mapreduce/lib/ > > > > > > snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > > jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > > paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > > > > > > guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > > > > > > jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > > > > > > hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > > > > > > mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > > > > > > hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > > > mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > > > > > > mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > > > > > > lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > > > > > > mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > > > > > > guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > > > > > > 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > > > > > > usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > > > > > > usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > > > > > > hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > > > > > > mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > > > > > > hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- > > > yarn-server- > > > > > > sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > > server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > > hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > > hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > > yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > > > > > > hadoop-yarn/hadoop-yarn-applications-distributedshell- > > > > > > 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > > server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- > > yarn-server- > > > > > > sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > > hadoop-yarn-server-applicationhistoryservice.jar: > > > > > > /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > > > > > > lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > > > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > > > > > > 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > > > applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > > yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > > > > > > lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > > > > > > yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > > yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > > > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > > > unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > > > > > > yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > > > > > > hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > > > > > > common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > > > > > > proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > > > > > > nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > > > > > > jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > > > > > > unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > > > > > > yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > > > > > > api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > > > > > > jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > > > > > > lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > > > > > > hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > > > > > > lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > > > > > > 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > > > > > > usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > > > > > > hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > > > > > > lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > > > > > > client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > > > > > > jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > > > > > > usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > > > > > > yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > > > > > > netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > > > > > > aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > > > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > > > > > > usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > > > > > > /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > > > > > > lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > > > > > > hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > > > > > > 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > > > > > > jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > > > > > > hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > > > > > > lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > > > collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > > > > > > 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > > > > > > usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > > > > > > hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > > > > > > lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > > > > > > jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > > > > > > logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > > > > > > /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > > > > > > share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > > > > > > jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > > > > > > bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > > > > > > over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > > > > > > java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > > > > > > 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > > > > > > usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > > > > > > share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > > > > > > share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > > > > > > usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > > > > > > lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > > > > > > share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > > > > > > aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > > > > > > aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > > > > > > usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > > > > > > gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > > java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > aws-java-sdk- > > > > > > elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk- > mechanicalturkrequester-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > > java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > > > > > > sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > > > > > > sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > stepfunctions-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > > java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > > > > > sdk/aws-java-sdk- > > > > > > cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > elastictranscoder-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > organizations-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > > > > > sdk/aws-java-sdk- > > > > > > applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > > > > > > aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > > java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > > java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk- > resourcegroupstaggingapi-1.11. > > > > > > 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > > > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > > > > > > java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > > > > > > usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > > > > > > /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > > > > > > jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > > > > > > 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > > > > > > java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > > > > > > aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > > > > > > share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > > > > > > 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > > > > > > elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > > > > > > aws-java-sdk-iot-1.11.160.jar > > > > > > > > > > > > Thank you > > > > > > > > > > > > On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > the CliFrontend is not executed via Yarn, thus, it is not > > affected > > > by > > > > > > > dependencies which are added due to the underlying Yarn > cluster. > > > > > > Therefore, > > > > > > > it would be helpful to look at the TaskManager logs. Either you > > > have > > > > > > > enabled log aggregation on your Yarn cluster, then you can > obtain > > > the > > > > > > logs > > > > > > > via `yarn logs -applicationId <application ID>` or you have to > > > > retrieve > > > > > > > them from the machines where they were running (either by going > > > > > directly > > > > > > > there or via the Yarn web interface). > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> > > wrote: > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > That is actually the classpath used by the flink bash > > script(that > > > > > > > launches > > > > > > > > the jar using the java command). I changed the execute to an > > > echo, > > > > > and > > > > > > > > grabbed that for the CLI arguments. > > > > > > > > > > > > > > > > I believe this is the class path from the log file(although > it > > > > might > > > > > > not > > > > > > > be > > > > > > > > the taskmanager log, is that any different from what would be > > in > > > my > > > > > > > > flink-1.3.2/log folder?): > > > > > > > > > > > > > > > > 2017-10-02 20:03:26,450 INFO org.apache.flink.client. > > > CliFrontend > > > > > > > > - Classpath: > > > > > > > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > > > > > > > > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > > > > > > > > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > > > > > > > > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > > > > > > > > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ > > > > hadoop/conf: > > > > > > > > > > > > > > > > If that doesn't seem right, and you can point me in the right > > > > > direction > > > > > > > as > > > > > > > > to where the TaskManager logs would be, I would be happy to > > grab > > > > the > > > > > > > > information your looking for. > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > this looks to me indeed like a dependency problem. I assume > > > that > > > > > EMR > > > > > > or > > > > > > > > > something else is pulling in an incompatible version of > > Hadoop. > > > > > > > > > > > > > > > > > > The classpath you've posted, is this the one logged in the > > log > > > > > files > > > > > > > > > (TaskManager log) or did you compile it yourself? In the > > latter > > > > > case, > > > > > > > it > > > > > > > > > would also be helpful to get access to the TaskManager > logs. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 10:20 PM, Andy M. < > [hidden email]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > > > 1) I have looked at the linked docs, and from what I can > > tell > > > > no > > > > > > > setup > > > > > > > > > > should really need to be done to get Flink working(Other > > than > > > > > > > > downloading > > > > > > > > > > the correct binaries, which I believe I did) > > > > > > > > > > 2) I have downloaded the Flink 1.3.2 > > > binaries(flink-1.3.2-bin- > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > <http://apache.claz.org/flink/ > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > hadoop27-scala_2.11.tgz>) > > > > > > > > > > This is for hadoop 2.7.X, which matches EMR 5.8.0. > > > > > > > > > > > > > > > > > > > > I appreciate any help or guidance you can provide me in > > > fixing > > > > my > > > > > > > > > problems, > > > > > > > > > > please let me know if there is anything else I can > provide > > > you. > > > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > > > I'm not an AWS expert, so I'll just check on some > common > > > > > issues. > > > > > > > > > > > > > > > > > > > > > > I guess you already had a look at the Flink docs for > > > AWS/EMR > > > > > but > > > > > > > I'll > > > > > > > > > > post > > > > > > > > > > > the link just be to sure [1]. > > > > > > > > > > > > > > > > > > > > > > Since you are using Flink 1.3.2 (EMR 5.8.0 comes with > > Flink > > > > > > 1.3.1) > > > > > > > > did > > > > > > > > > > you > > > > > > > > > > > built Flink yourself or did you download the binaries? > > > > > > > > > > > Does the Hadoop version of the Flink build match the > > Hadoop > > > > > > version > > > > > > > > of > > > > > > > > > > EMR > > > > > > > > > > > 5.8.0, i.e., Hadoop 2.7.x? > > > > > > > > > > > > > > > > > > > > > > Best, Fabian > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs- > > > > > > > > > > release-1.3/setup/aws.html > > > > > > > > > > > > > > > > > > > > > > 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email] > >: > > > > > > > > > > > > > > > > > > > > > > > Hi Fabian, > > > > > > > > > > > > > > > > > > > > > > > > Sorry, I just realized I forgot to include that part. > > > The > > > > > > error > > > > > > > > > > returned > > > > > > > > > > > > is: > > > > > > > > > > > > > > > > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > > org.apache.hadoop.conf.Configuration.addResource( > > > > > > > > > > > Lorg/apache/hadoop/conf/ > > > > > > > > > > > > Configuration;)V > > > > > > > > > > > > at com.amazon.ws.emr.hadoop.fs. > > > > EmrFileSystem.initialize( > > > > > > > > > > > > EmrFileSystem.java:93) > > > > > > > > > > > > at org.apache.flink.runtime.fs. > > > hdfs.HadoopFileSystem. > > > > > > > > > > > > initialize(HadoopFileSystem.java:328) > > > > > > > > > > > > at org.apache.flink.core.fs.FileSystem. > > > > > > > getUnguardedFileSystem( > > > > > > > > > > > > FileSystem.java:350) > > > > > > > > > > > > at org.apache.flink.core.fs. > > > FileSystem.get(FileSystem. > > > > > > > > java:389) > > > > > > > > > > > > at org.apache.flink.core.fs.Path. > > > > > > > getFileSystem(Path.java:293) > > > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > > > > > > > FsCheckpointStreamFactory.<init>( > > > > FsCheckpointStreamFactory. > > > > > > > > java:99) > > > > > > > > > > > > at org.apache.flink.runtime.state.filesystem. > > > > > > FsStateBackend. > > > > > > > > > > > > createStreamFactory(FsStateBackend.java:282) > > > > > > > > > > > > at org.apache.flink.contrib.streaming.state. > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > > createStreamFactory(RocksDBStateBackend.java:273 > > > > > > > > > > > > > > > > > > > > > > > > I believe it has something to do with the classpath, > > but > > > I > > > > am > > > > > > > > unsure > > > > > > > > > > why > > > > > > > > > > > or > > > > > > > > > > > > how to fix it. The classpath being used during the > > > > execution > > > > > > is: > > > > > > > > > > > > /home/hadoop/flink-1.3.2/lib/ > > > flink-python_2.11-1.3.2.jar:/ > > > > > > > > > > > > home/hadoop/flink-1.3.2/ > > lib/flink-shaded-hadoop2- > > > > > > > > > > > > uber-1.3.2.jar:/home/hadoop/ > > > > flink-1.3.2/lib/log4j-1.2. > > > > > > > > > > > > 17.jar:/home/hadoop/flink-1. > > > > 3.2/lib/slf4j-log4j12-1.7.7. > > > > > > > > > > > > jar:/home/hadoop/flink-1. > > > > 3.2/lib/flink-dist_2.11-1.3. > > > > > > > > > > > > 2.jar::/etc/hadoop/conf: > > > > > > > > > > > > > > > > > > > > > > > > I decompiled flink-shaded-hadoop2-uber-1.3.2.jar > > and > > > > it > > > > > > > seems > > > > > > > > > the > > > > > > > > > > > > addResource function does seem to be there. > > > > > > > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > > > > > > > can you describe in more detail what exactly isn't > > > > working? > > > > > > > > > > > > > Do you see error messages in the log files or on > the > > > > > console? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Fabian > > > > > > > > > > > > > > > > > > > > > > > > > > 2017-10-02 15:52 GMT+02:00 Andy M. < > > [hidden email] > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am about to deploy my first Flink projects to > > > > > > production, > > > > > > > > but > > > > > > > > > I > > > > > > > > > > am > > > > > > > > > > > > > > running into a very big hurdle. I am unable to > > > launch > > > > my > > > > > > > > project > > > > > > > > > > so > > > > > > > > > > > it > > > > > > > > > > > > > can > > > > > > > > > > > > > > write to an S3 bucket. My project is running on > an > > > EMR > > > > > > > > cluster, > > > > > > > > > > > where > > > > > > > > > > > > I > > > > > > > > > > > > > > have installed Flink 1.3.2. I am using Yarn to > > > launch > > > > > the > > > > > > > > > > > application, > > > > > > > > > > > > > and > > > > > > > > > > > > > > it seems to run fine unless I am trying to enable > > > check > > > > > > > > > > > pointing(with a > > > > > > > > > > > > > S3 > > > > > > > > > > > > > > target). I am looking to use RocksDB as my > > > > > check-pointing > > > > > > > > > backend. > > > > > > > > > > > I > > > > > > > > > > > > > have > > > > > > > > > > > > > > asked a few places, and I am still unable to > find a > > > > > > solution > > > > > > > to > > > > > > > > > > this > > > > > > > > > > > > > > problem. Here are my steps for creating a > cluster, > > > and > > > > > > > > launching > > > > > > > > > > my > > > > > > > > > > > > > > application, perhaps I am missing a step. I'd be > > > happy > > > > > to > > > > > > > > > provide > > > > > > > > > > > any > > > > > > > > > > > > > > additional information if needed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > AWS Portal: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) EMR -> Create Cluster > > > > > > > > > > > > > > 2) Advanced Options > > > > > > > > > > > > > > 3) Release = emr-5.8.0 > > > > > > > > > > > > > > 4) Only select Hadoop 2.7.3 > > > > > > > > > > > > > > 5) Next -> Next -> Next -> Create Cluster ( I > > do > > > > fill > > > > > > out > > > > > > > > > > > > > > names/keys/etc) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Once the cluster is up I ssh into the Master and > do > > > the > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1 wget > > > > > > > > > > > > > > http://apache.claz.org/flink/ > > > > > flink-1.3.2/flink-1.3.2-bin- > > > > > > > > > > > > > > hadoop27-scala_2.11.tgz > > > > > > > > > > > > > > 2 tar -xzf flink-1.3.2-bin-hadoop27- > > > > scala_2.11.tgz > > > > > > > > > > > > > > 3 cd flink-1.3.2 > > > > > > > > > > > > > > 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 > -d > > > > > > > > > > > > > > 5 Change conf/flink-conf.yaml > > > > > > > > > > > > > > 6 ./bin/flink run -m yarn-cluster -yn 1 > > > > > > > > ~/flink-consumer.jar > > > > > > > > > > > > > > > > > > > > > > > > > > > > My conf/flink-conf.yaml I add the following > fields: > > > > > > > > > > > > > > > > > > > > > > > > > > > > state.backend: rocksdb > > > > > > > > > > > > > > state.backend.fs.checkpointdir: > > > > s3:/bucket/location > > > > > > > > > > > > > > state.checkpoints.dir: s3:/bucket/location > > > > > > > > > > > > > > > > > > > > > > > > > > > > My program's checkpointing setup: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.enableCheckpointing(getCheckpointRate, > > > > > > > > > > CheckpointingMode.EXACTLY_ > > > > > > > > > > > > > ONCE) > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > > > enableExternalizedCheckpoints( > > > > > > > > > > > > > > ExternalizedCheckpointCleanup. > > > RETAIN_ON_CANCELLATION) > > > > > > > > > > > > > > > > > > > > > > > > > > > > env.getCheckpointConfig. > > > setMinPauseBetweenCheckpoints( > > > > > > > > > > > > > > getCheckpointMinPause) > > > > > > > > > > > > > > env.getCheckpointConfig. > setCheckpointTimeout( > > > > > > > > > > > getCheckpointTimeout) > > > > > > > > > > > > > > env.getCheckpointConfig. > > > > > setMaxConcurrentCheckpoints(1) > > > > > > > > > > > > > > env.setStateBackend(new > > > RocksDBStateBackend("s3:// > > > > > > > > > > > > bucket/location", > > > > > > > > > > > > > > true)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Hi Andy,
I remember that I was testing a job with almost exactly the same setup as part of the Flink 1.3.2 release testing. The command I used to start my job is roughly this: HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m yarn-cluster -yn 5 -ys 2 ... i.e. I export the proper hadoop config dir and I run a per-job YARN cluster. I think I also exported the result of "hadoop classpath" as HADOOP_CLASSPATH. Best, Aljoscha > On 10. Oct 2017, at 16:43, Andy M. <[hidden email]> wrote: > > Hello, > > Bowen: Unless I am missing something, it says there needs to be no setup > on EMR, Each topic says: "You don’t have to configure this manually if you > are running Flink on EMR." S3 access from CLI works fine on my clusters. > > Chen: Thank you for this, I will look into this if I am unable to get this > running on YARN successfully. > > Stephan: Removing the said library causes the flink > (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs > this to work. I tried explicitly setting the classpath for the java call > as well to point to the hadoop library jars. This is the original java > command that I was trying to run: > > java > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-ip-172-31-19-27.log > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/log4j-cli.properties > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/conf/logback.xml > -classpath > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 > /home/hadoop/flink-consumer.jar > > > This is what I changed it too(removing the shadded-hadoop2-uber jar and > adding in the hadoop folder): > > java > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-ip-172-31-19-27.log > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/log4j-cli.properties > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/conf/logback.xml > -classpath > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 > /home/hadoop/flink-consumer.jar > > The later throws the following error: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/util/VersionInfo > at > org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:283) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1124) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.util.VersionInfo > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 2 more > > Simply removing the .jar from the folder causes the same error. > > Thank you > > > > > On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <[hidden email]> wrote: > >> Hi! >> >> It looks like multiple Hadoop versions are in the classpath. Flink's hadoop >> jar and the EMR Hadoop jars. >> I would simply drop Flink's own Hadoop dependency and only use the EMR >> Hadoop jars. >> >> Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the >> setup is such that the Hadoop lib environment variable is set. Then it >> should not have conflicts any more. >> >> >> >> On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <[hidden email]> wrote: >> >>> Attached my side project verified working to deploy jobmanager and >>> taskmanager as stateless service(non yarn/mesos), configuration here >>> >>> https://github.com/chenqin/flink-jar/tree/master/config/hadoop >>> >>> more detail here >>> https://github.com/chenqin/flink-jar/blob/master/src/ >>> main/java/FlinkBootstrap.java#L49 >>> >>> On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <[hidden email]> >> wrote: >>> >>>> Hi Andy, >>>> >>>> I believe it's because you didn't set your s3 impl correctly. Try to >> set >>>> your core-site.xml by following https://ci.apache.org/ >>>> projects/flink/flink-docs-release-1.4/ops/deployment/ >>>> aws.html#s3afilesystem- >>>> recommended >>>> >>>> Bowen >>>> >>>> On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: >>>> >>>>> Hi Till, >>>>> >>>>> Seems like everything is in line there. hadoop-common.jar -> >>>>> hadoop-common-2.7.3-amzn-3.jar >>>>> >>>>> And when i decompiled that jar I see public void >>>> addResource(Configuration >>>>> conf) in org/apache/hadoop/conf/Configuration.java >>>>> >>>>> I agree that an incorrect version of the jar is probably being run, >> is >>>>> there a way to limit the classpath for the TaskManager when starting >>> the >>>>> job? >>>>> >>>>> Thank you >>>>> >>>>> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> >>>>> wrote: >>>>> >>>>>> Hi Andy, >>>>>> >>>>>> could you check which Hadoop version this jar >>>>>> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether >>> the >>>>>> contained hadoop Configuration class has the method >>>>>> Configuration.addResource(Lorg/apache/hadoop/conf/ >> Configuration;)V. >>>>> Maybe >>>>>> this jar is the culprit because it comes from a different Hadoop >>>> version. >>>>>> >>>>>> Cheers, >>>>>> Till >>>>>> >>>>>> >>>>>> On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: >>>>>> >>>>>>> Hi Till, >>>>>>> >>>>>>> I believe this is what you are looking for, classpath is much >>> bigger >>>>> for >>>>>>> the task manager. I can also post the whole log file if needed: >>>>>>> >>>>>>> 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. >>>>>> YarnTaskManagerRunner >>>>>>> - Classpath: >>>>>>> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- >>>>>>> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. >>>>>>> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: >>>>>>> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ >>>>>>> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- >>>>>>> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ >>>>>>> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- >>>>>>> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ >>>>>>> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ >>>>>>> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ >>>>>>> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ >>>>>>> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ >>>>>>> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. >>>>>>> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ >>>>>>> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ >>>>>>> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. >>>>>>> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. >>>>>>> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ >>>>>>> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ >>>>>>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. >>>>>>> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. >>>>>>> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ >>>>>>> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- >>>>>>> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ >>>>>>> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ >>>>>>> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. >>>>>>> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ >>>>>>> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ >>>>>>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- >>>>>>> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. >>>>>>> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ >>>>>>> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ >>>>>>> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ >>>>>>> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. >>>>>>> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ >>>>>>> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. >>>>>>> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ >>>>>>> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ >>>>>>> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. >>>>>>> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ >>>>>>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- >>>>>>> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- >>>>>>> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: >>>>>>> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ >>>>>>> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- >>>>>>> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ >>>>>>> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ >>>>>>> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ >>>>>>> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. >>>>>>> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: >>>>>>> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ >>>>>>> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. >>>>>>> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ >>>>>>> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ >>>>>>> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. >>>>>>> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ >>>>>>> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ >>>>>>> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. >>>>>>> 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ >>>>>>> hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- >>>>>>> incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- >>>>>>> 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ >>>>>>> lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ >>>>>>> lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ >>>>>>> commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ >>>>>>> usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- >>>>>>> 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ >>>>>>> lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- >>>>>>> hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ >>>>>>> lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- >>>>>>> api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: >>>>>>> /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. >>>>>>> jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- >>>>>>> mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ >>>>>> hadoop-mapreduce/hadoop- >>>>>>> mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- >>>>>>> mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- >>>>>>> mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- >>>>>>> mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. >>>>>>> jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. >>>>>> 8.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. >>>>>>> jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- >>>>>>> net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- >>>>>>> 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- >>>>>>> mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- >>>>>>> mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- >>>>>>> mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- >>>>>>> mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- >>>>>>> mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- >>>>>>> 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. >>>>>>> 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- >>>>>>> client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. >>>>>>> jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- >>>>>>> common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- >>>>>>> jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- >>>>>>> mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- >>>>>>> archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. >>>>>>> 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. >>>>>>> jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- >>>>>>> mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- >>>>>>> kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. >>>>>>> jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: >>>>>>> /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- >>>> mapreduce/commons- >>>>>>> configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- >>>>>>> 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- >>>>>>> 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: >>>>>>> /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ >>>>>> lib/hadoop-mapreduce/lib/ >>>>>>> snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>> jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>> paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- >>>>>>> guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. >>>>>>> jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- >>>>>>> mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>> guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. >>>>>>> 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- >>>> yarn-server- >>>>>>> sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>> server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ >>>>>>> hadoop-yarn/hadoop-yarn-applications-distributedshell- >>>>>>> 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>> server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- >>> yarn-server- >>>>>>> sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-server-applicationhistoryservice.jar: >>>>>>> /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ >>>>>>> lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- >>>>>>> applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- >>>>>>> unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- >>>>>>> yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>> common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- >>>>>>> proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- >>>>>>> nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- >>>>>>> unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- >>>>>>> api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. >>>>>>> jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ >>>>>>> lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. >>>>>>> 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- >>>>>>> client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- >>>>>>> yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ >>>>>>> aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: >>>>>>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ >>>>>>> lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- >>>>>>> 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- >>>>>>> collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- >>>>>>> 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: >>>>>>> /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ >>>>>>> share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ >>>>>>> jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ >>>>>>> bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- >>>>>>> over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- >>>>>>> java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. >>>>>>> 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ >>>>>>> usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ >>>>>>> share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ >>>>>>> share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ >>>>>>> usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ >>>>>>> lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ >>>>>>> share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ >>>>>>> aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ >>>>>>> aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ >>>>>>> usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- >>>>>>> gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>> aws-java-sdk- >>>>>>> elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk- >> mechanicalturkrequester-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>> sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>> sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>> stepfunctions-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- >>>>>> sdk/aws-java-sdk- >>>>>>> cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>> elastictranscoder-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>> organizations-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- >>>>>> sdk/aws-java-sdk- >>>>>>> applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ >>>>>>> aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk- >> resourcegroupstaggingapi-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-iot-1.11.160.jar >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < >>> [hidden email]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Andy, >>>>>>>> >>>>>>>> the CliFrontend is not executed via Yarn, thus, it is not >>> affected >>>> by >>>>>>>> dependencies which are added due to the underlying Yarn >> cluster. >>>>>>> Therefore, >>>>>>>> it would be helpful to look at the TaskManager logs. Either you >>>> have >>>>>>>> enabled log aggregation on your Yarn cluster, then you can >> obtain >>>> the >>>>>>> logs >>>>>>>> via `yarn logs -applicationId <application ID>` or you have to >>>>> retrieve >>>>>>>> them from the machines where they were running (either by going >>>>>> directly >>>>>>>> there or via the Yarn web interface). >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>>> >>>>>>>> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> >>> wrote: >>>>>>>> >>>>>>>>> Hi Till, >>>>>>>>> >>>>>>>>> That is actually the classpath used by the flink bash >>> script(that >>>>>>>> launches >>>>>>>>> the jar using the java command). I changed the execute to an >>>> echo, >>>>>> and >>>>>>>>> grabbed that for the CLI arguments. >>>>>>>>> >>>>>>>>> I believe this is the class path from the log file(although >> it >>>>> might >>>>>>> not >>>>>>>> be >>>>>>>>> the taskmanager log, is that any different from what would be >>> in >>>> my >>>>>>>>> flink-1.3.2/log folder?): >>>>>>>>> >>>>>>>>> 2017-10-02 20:03:26,450 INFO org.apache.flink.client. >>>> CliFrontend >>>>>>>>> - Classpath: >>>>>>>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ >>>>>>>>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. >>>>>>>>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ >>>>>>>>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ >>>>>>>>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ >>>>> hadoop/conf: >>>>>>>>> >>>>>>>>> If that doesn't seem right, and you can point me in the right >>>>>> direction >>>>>>>> as >>>>>>>>> to where the TaskManager logs would be, I would be happy to >>> grab >>>>> the >>>>>>>>> information your looking for. >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < >>>>> [hidden email]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Andy, >>>>>>>>>> >>>>>>>>>> this looks to me indeed like a dependency problem. I assume >>>> that >>>>>> EMR >>>>>>> or >>>>>>>>>> something else is pulling in an incompatible version of >>> Hadoop. >>>>>>>>>> >>>>>>>>>> The classpath you've posted, is this the one logged in the >>> log >>>>>> files >>>>>>>>>> (TaskManager log) or did you compile it yourself? In the >>> latter >>>>>> case, >>>>>>>> it >>>>>>>>>> would also be helpful to get access to the TaskManager >> logs. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Till >>>>>>>>>> >>>>>>>>>> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. < >> [hidden email]> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Fabian, >>>>>>>>>>> >>>>>>>>>>> 1) I have looked at the linked docs, and from what I can >>> tell >>>>> no >>>>>>>> setup >>>>>>>>>>> should really need to be done to get Flink working(Other >>> than >>>>>>>>> downloading >>>>>>>>>>> the correct binaries, which I believe I did) >>>>>>>>>>> 2) I have downloaded the Flink 1.3.2 >>>> binaries(flink-1.3.2-bin- >>>>>>>>>>> hadoop27-scala_2.11.tgz >>>>>>>>>>> <http://apache.claz.org/flink/ >> flink-1.3.2/flink-1.3.2-bin- >>>>>>>>>>> hadoop27-scala_2.11.tgz>) >>>>>>>>>>> This is for hadoop 2.7.X, which matches EMR 5.8.0. >>>>>>>>>>> >>>>>>>>>>> I appreciate any help or guidance you can provide me in >>>> fixing >>>>> my >>>>>>>>>> problems, >>>>>>>>>>> please let me know if there is anything else I can >> provide >>>> you. >>>>>>>>>>> >>>>>>>>>>> Thank you >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < >>>>> [hidden email] >>>>>>> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Andy, >>>>>>>>>>>> >>>>>>>>>>>> I'm not an AWS expert, so I'll just check on some >> common >>>>>> issues. >>>>>>>>>>>> >>>>>>>>>>>> I guess you already had a look at the Flink docs for >>>> AWS/EMR >>>>>> but >>>>>>>> I'll >>>>>>>>>>> post >>>>>>>>>>>> the link just be to sure [1]. >>>>>>>>>>>> >>>>>>>>>>>> Since you are using Flink 1.3.2 (EMR 5.8.0 comes with >>> Flink >>>>>>> 1.3.1) >>>>>>>>> did >>>>>>>>>>> you >>>>>>>>>>>> built Flink yourself or did you download the binaries? >>>>>>>>>>>> Does the Hadoop version of the Flink build match the >>> Hadoop >>>>>>> version >>>>>>>>> of >>>>>>>>>>> EMR >>>>>>>>>>>> 5.8.0, i.e., Hadoop 2.7.x? >>>>>>>>>>>> >>>>>>>>>>>> Best, Fabian >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs- >>>>>>>>>>> release-1.3/setup/aws.html >>>>>>>>>>>> >>>>>>>>>>>> 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email] >>> : >>>>>>>>>>>> >>>>>>>>>>>>> Hi Fabian, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry, I just realized I forgot to include that part. >>>> The >>>>>>> error >>>>>>>>>>> returned >>>>>>>>>>>>> is: >>>>>>>>>>>>> >>>>>>>>>>>>> java.lang.NoSuchMethodError: >>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.addResource( >>>>>>>>>>>> Lorg/apache/hadoop/conf/ >>>>>>>>>>>>> Configuration;)V >>>>>>>>>>>>> at com.amazon.ws.emr.hadoop.fs. >>>>> EmrFileSystem.initialize( >>>>>>>>>>>>> EmrFileSystem.java:93) >>>>>>>>>>>>> at org.apache.flink.runtime.fs. >>>> hdfs.HadoopFileSystem. >>>>>>>>>>>>> initialize(HadoopFileSystem.java:328) >>>>>>>>>>>>> at org.apache.flink.core.fs.FileSystem. >>>>>>>> getUnguardedFileSystem( >>>>>>>>>>>>> FileSystem.java:350) >>>>>>>>>>>>> at org.apache.flink.core.fs. >>>> FileSystem.get(FileSystem. >>>>>>>>> java:389) >>>>>>>>>>>>> at org.apache.flink.core.fs.Path. >>>>>>>> getFileSystem(Path.java:293) >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. >>>>>>>>>>>>> FsCheckpointStreamFactory.<init>( >>>>> FsCheckpointStreamFactory. >>>>>>>>> java:99) >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. >>>>>>> FsStateBackend. >>>>>>>>>>>>> createStreamFactory(FsStateBackend.java:282) >>>>>>>>>>>>> at org.apache.flink.contrib.streaming.state. >>>>>>>>> RocksDBStateBackend. >>>>>>>>>>>>> createStreamFactory(RocksDBStateBackend.java:273 >>>>>>>>>>>>> >>>>>>>>>>>>> I believe it has something to do with the classpath, >>> but >>>> I >>>>> am >>>>>>>>> unsure >>>>>>>>>>> why >>>>>>>>>>>> or >>>>>>>>>>>>> how to fix it. The classpath being used during the >>>>> execution >>>>>>> is: >>>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/ >>>> flink-python_2.11-1.3.2.jar:/ >>>>>>>>>>>>> home/hadoop/flink-1.3.2/ >>> lib/flink-shaded-hadoop2- >>>>>>>>>>>>> uber-1.3.2.jar:/home/hadoop/ >>>>> flink-1.3.2/lib/log4j-1.2. >>>>>>>>>>>>> 17.jar:/home/hadoop/flink-1. >>>>> 3.2/lib/slf4j-log4j12-1.7.7. >>>>>>>>>>>>> jar:/home/hadoop/flink-1. >>>>> 3.2/lib/flink-dist_2.11-1.3. >>>>>>>>>>>>> 2.jar::/etc/hadoop/conf: >>>>>>>>>>>>> >>>>>>>>>>>>> I decompiled flink-shaded-hadoop2-uber-1.3.2.jar >>> and >>>>> it >>>>>>>> seems >>>>>>>>>> the >>>>>>>>>>>>> addResource function does seem to be there. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < >>>>>>> [hidden email] >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Andy, >>>>>>>>>>>>>> >>>>>>>>>>>>>> can you describe in more detail what exactly isn't >>>>> working? >>>>>>>>>>>>>> Do you see error messages in the log files or on >> the >>>>>> console? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Fabian >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-10-02 15:52 GMT+02:00 Andy M. < >>> [hidden email] >>>>> : >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am about to deploy my first Flink projects to >>>>>>> production, >>>>>>>>> but >>>>>>>>>> I >>>>>>>>>>> am >>>>>>>>>>>>>>> running into a very big hurdle. I am unable to >>>> launch >>>>> my >>>>>>>>> project >>>>>>>>>>> so >>>>>>>>>>>> it >>>>>>>>>>>>>> can >>>>>>>>>>>>>>> write to an S3 bucket. My project is running on >> an >>>> EMR >>>>>>>>> cluster, >>>>>>>>>>>> where >>>>>>>>>>>>> I >>>>>>>>>>>>>>> have installed Flink 1.3.2. I am using Yarn to >>>> launch >>>>>> the >>>>>>>>>>>> application, >>>>>>>>>>>>>> and >>>>>>>>>>>>>>> it seems to run fine unless I am trying to enable >>>> check >>>>>>>>>>>> pointing(with a >>>>>>>>>>>>>> S3 >>>>>>>>>>>>>>> target). I am looking to use RocksDB as my >>>>>> check-pointing >>>>>>>>>> backend. >>>>>>>>>>>> I >>>>>>>>>>>>>> have >>>>>>>>>>>>>>> asked a few places, and I am still unable to >> find a >>>>>>> solution >>>>>>>> to >>>>>>>>>>> this >>>>>>>>>>>>>>> problem. Here are my steps for creating a >> cluster, >>>> and >>>>>>>>> launching >>>>>>>>>>> my >>>>>>>>>>>>>>> application, perhaps I am missing a step. I'd be >>>> happy >>>>>> to >>>>>>>>>> provide >>>>>>>>>>>> any >>>>>>>>>>>>>>> additional information if needed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AWS Portal: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) EMR -> Create Cluster >>>>>>>>>>>>>>> 2) Advanced Options >>>>>>>>>>>>>>> 3) Release = emr-5.8.0 >>>>>>>>>>>>>>> 4) Only select Hadoop 2.7.3 >>>>>>>>>>>>>>> 5) Next -> Next -> Next -> Create Cluster ( I >>> do >>>>> fill >>>>>>> out >>>>>>>>>>>>>>> names/keys/etc) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Once the cluster is up I ssh into the Master and >> do >>>> the >>>>>>>>>> following: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1 wget >>>>>>>>>>>>>>> http://apache.claz.org/flink/ >>>>>> flink-1.3.2/flink-1.3.2-bin- >>>>>>>>>>>>>>> hadoop27-scala_2.11.tgz >>>>>>>>>>>>>>> 2 tar -xzf flink-1.3.2-bin-hadoop27- >>>>> scala_2.11.tgz >>>>>>>>>>>>>>> 3 cd flink-1.3.2 >>>>>>>>>>>>>>> 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 >> -d >>>>>>>>>>>>>>> 5 Change conf/flink-conf.yaml >>>>>>>>>>>>>>> 6 ./bin/flink run -m yarn-cluster -yn 1 >>>>>>>>> ~/flink-consumer.jar >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My conf/flink-conf.yaml I add the following >> fields: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> state.backend: rocksdb >>>>>>>>>>>>>>> state.backend.fs.checkpointdir: >>>>> s3:/bucket/location >>>>>>>>>>>>>>> state.checkpoints.dir: s3:/bucket/location >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My program's checkpointing setup: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> env.enableCheckpointing(getCheckpointRate, >>>>>>>>>>> CheckpointingMode.EXACTLY_ >>>>>>>>>>>>>> ONCE) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> env.getCheckpointConfig. >>>> enableExternalizedCheckpoints( >>>>>>>>>>>>>>> ExternalizedCheckpointCleanup. >>>> RETAIN_ON_CANCELLATION) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> env.getCheckpointConfig. >>>> setMinPauseBetweenCheckpoints( >>>>>>>>>>>>>>> getCheckpointMinPause) >>>>>>>>>>>>>>> env.getCheckpointConfig. >> setCheckpointTimeout( >>>>>>>>>>>> getCheckpointTimeout) >>>>>>>>>>>>>>> env.getCheckpointConfig. >>>>>> setMaxConcurrentCheckpoints(1) >>>>>>>>>>>>>>> env.setStateBackend(new >>>> RocksDBStateBackend("s3:// >>>>>>>>>>>>> bucket/location", >>>>>>>>>>>>>>> true)) >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> |
Hi Ajoscha,
That didn't seem to do the trick either. Do the following look correct? I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same problem, all I did was upload my Scala .jar to the master, updated my flink-conf.yaml, set my env variables, and ran it with the following command: HADOOP_CONF_DIR=/etc/hadoop/conf flink run -m yarn-cluster -yn 5 -ys 2 ~/flink-consumer.jar I am beginning to think something else may be wrong with my configuration. Does the following look correct? In flink-conf.yaml: state.backend: rocksdb state.backend.fs.checkpointdir: s3://org/flink-project/state state.checkpoints.dir: s3://org/flink-project/state In the code: env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE) env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) env.getCheckpointConfig.setMinPauseBetweenCheckpoints(1000) env.getCheckpointConfig.setCheckpointTimeout(5000) env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) env.setStateBackend(new RocksDBStateBackend("s3://org/flink-project/state", true)) Thank you On Wed, Oct 11, 2017 at 5:19 AM, Aljoscha Krettek <[hidden email]> wrote: > Hi Andy, > > I remember that I was testing a job with almost exactly the same setup as > part of the Flink 1.3.2 release testing. The command I used to start my job > is roughly this: > > HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m > yarn-cluster -yn 5 -ys 2 ... > > i.e. I export the proper hadoop config dir and I run a per-job YARN > cluster. I think I also exported the result of "hadoop classpath" as > HADOOP_CLASSPATH. > > Best, > Aljoscha > > > On 10. Oct 2017, at 16:43, Andy M. <[hidden email]> wrote: > > > > Hello, > > > > Bowen: Unless I am missing something, it says there needs to be no setup > > on EMR, Each topic says: "You don’t have to configure this manually if > you > > are running Flink on EMR." S3 access from CLI works fine on my clusters. > > > > Chen: Thank you for this, I will look into this if I am unable to get > this > > running on YARN successfully. > > > > Stephan: Removing the said library causes the flink > > (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs > > this to work. I tried explicitly setting the classpath for the java call > > as well to point to the hadoop library jars. This is the original java > > command that I was trying to run: > > > > java > > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client- > ip-172-31-19-27.log > > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/ > log4j-cli.properties > > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/ > conf/logback.xml > > -classpath > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 > > /home/hadoop/flink-consumer.jar > > > > > > This is what I changed it too(removing the shadded-hadoop2-uber jar and > > adding in the hadoop folder): > > > > java > > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client- > ip-172-31-19-27.log > > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/ > log4j-cli.properties > > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/ > conf/logback.xml > > -classpath > > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/ > flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink- > 1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/ > activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar: > /usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/ > apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/ > usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/ > lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/ > lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java- > xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final. > jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/ > lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/ > jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/ > usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2. > 3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator- > client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9. > jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/ > hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator- > framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/ > hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/ > curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey- > server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7. > 10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/ > jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > 1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/ > hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1. > 1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/ > hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26- > emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons- > collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3. > 1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26- > emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/ > lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/ > httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/ > usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/ > lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/ > httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/ > usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/ > hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/ > jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient- > 3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/ > lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf > > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 > > /home/hadoop/flink-consumer.jar > > > > The later throws the following error: > > > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > > [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. > jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > > [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/ > slf4j/impl/StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > explanation. > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > Exception in thread "main" java.lang.NoClassDefFoundError: > > org/apache/hadoop/util/VersionInfo > > at > > org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo( > EnvironmentInformation.java:283) > > at org.apache.flink.client.CliFrontend.main(CliFrontend. > java:1124) > > Caused by: java.lang.ClassNotFoundException: > > org.apache.hadoop.util.VersionInfo > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 2 more > > > > Simply removing the .jar from the folder causes the same error. > > > > Thank you > > > > > > > > > > On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <[hidden email]> wrote: > > > >> Hi! > >> > >> It looks like multiple Hadoop versions are in the classpath. Flink's > hadoop > >> jar and the EMR Hadoop jars. > >> I would simply drop Flink's own Hadoop dependency and only use the EMR > >> Hadoop jars. > >> > >> Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure > the > >> setup is such that the Hadoop lib environment variable is set. Then it > >> should not have conflicts any more. > >> > >> > >> > >> On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <[hidden email]> wrote: > >> > >>> Attached my side project verified working to deploy jobmanager and > >>> taskmanager as stateless service(non yarn/mesos), configuration here > >>> > >>> https://github.com/chenqin/flink-jar/tree/master/config/hadoop > >>> > >>> more detail here > >>> https://github.com/chenqin/flink-jar/blob/master/src/ > >>> main/java/FlinkBootstrap.java#L49 > >>> > >>> On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <[hidden email]> > >> wrote: > >>> > >>>> Hi Andy, > >>>> > >>>> I believe it's because you didn't set your s3 impl correctly. Try to > >> set > >>>> your core-site.xml by following https://ci.apache.org/ > >>>> projects/flink/flink-docs-release-1.4/ops/deployment/ > >>>> aws.html#s3afilesystem- > >>>> recommended > >>>> > >>>> Bowen > >>>> > >>>> On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: > >>>> > >>>>> Hi Till, > >>>>> > >>>>> Seems like everything is in line there. hadoop-common.jar -> > >>>>> hadoop-common-2.7.3-amzn-3.jar > >>>>> > >>>>> And when i decompiled that jar I see public void > >>>> addResource(Configuration > >>>>> conf) in org/apache/hadoop/conf/Configuration.java > >>>>> > >>>>> I agree that an incorrect version of the jar is probably being run, > >> is > >>>>> there a way to limit the classpath for the TaskManager when starting > >>> the > >>>>> job? > >>>>> > >>>>> Thank you > >>>>> > >>>>> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> > >>>>> wrote: > >>>>> > >>>>>> Hi Andy, > >>>>>> > >>>>>> could you check which Hadoop version this jar > >>>>>> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether > >>> the > >>>>>> contained hadoop Configuration class has the method > >>>>>> Configuration.addResource(Lorg/apache/hadoop/conf/ > >> Configuration;)V. > >>>>> Maybe > >>>>>> this jar is the culprit because it comes from a different Hadoop > >>>> version. > >>>>>> > >>>>>> Cheers, > >>>>>> Till > >>>>>> > >>>>>> > >>>>>> On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: > >>>>>> > >>>>>>> Hi Till, > >>>>>>> > >>>>>>> I believe this is what you are looking for, classpath is much > >>> bigger > >>>>> for > >>>>>>> the task manager. I can also post the whole log file if needed: > >>>>>>> > >>>>>>> 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. > >>>>>> YarnTaskManagerRunner > >>>>>>> - Classpath: > >>>>>>> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- > >>>>>>> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. > >>>>>>> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: > >>>>>>> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ > >>>>>>> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- > >>>>>>> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. > >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > >>>>>>> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- > >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- > >>>>>>> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ > >>>>>>> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. > >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ > >>>>>>> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. > >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- > >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. > >>>>>>> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ > >>>>>>> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > >>>>>>> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- > >>>>>>> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ > >>>>>>> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- > >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. > >>>>>>> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. > >>>>>>> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ > >>>>>>> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ > >>>>>>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. > >>>>>>> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. > >>>>>>> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ > >>>>>>> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- > >>>>>>> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ > >>>>>>> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ > >>>>>>> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. > >>>>>>> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ > >>>>>>> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ > >>>>>>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- > >>>>>>> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. > >>>>>>> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ > >>>>>>> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ > >>>>>>> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ > >>>>>>> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ > >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. > >>>>>>> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ > >>>>>>> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. > >>>>>>> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ > >>>>>>> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ > >>>>>>> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. > >>>>>>> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ > >>>>>>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- > >>>>>>> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- > >>>>>>> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: > >>>>>>> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ > >>>>>>> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- > >>>>>>> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ > >>>>>>> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ > >>>>>>> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ > >>>>>>> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. > >>>>>>> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: > >>>>>>> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ > >>>>>>> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. > >>>>>>> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ > >>>>>>> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ > >>>>>>> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. > >>>>>>> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ > >>>>>>> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ > >>>>>>> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- > >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. > >>>>>>> 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ > >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ > >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ > >>>>>>> hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- > >>>>>>> amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- > >>>>>>> incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- > >>>>>>> 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. > >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ > >>>>>>> lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ > >>>>>>> lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ > >>>>>>> commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. > >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ > >>>>>>> usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ > >>>>>>> hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ > >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- > >>>>>>> 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ > >>>>>>> hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ > >>>>>>> lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ > >>>>>>> hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > >>>>>>> hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ > >>>>>>> lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- > >>>>>>> api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: > >>>>>>> /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ > >>>>>>> hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ > >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- > >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ > >>>>>> hadoop-mapreduce/hadoop- > >>>>>>> mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- > >>>>>>> mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- > >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. > >>>>>> 8.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- > >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- > >>>>>>> net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- > >>>>>>> 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- > >>>>>>> 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: > >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. > >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. > >>>>>>> 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- > >>>>>>> client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > >>>>>>> common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. > >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- > >>>>>>> jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- > >>>>>>> archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. > >>>>>>> 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- > >>>>>>> kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: > >>>>>>> /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: > >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- > >>>> mapreduce/commons- > >>>>>>> configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- > >>>>>>> 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- > >>>>>>> 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: > >>>>>>> /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ > >>>>>>> lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ > >>>>>> lib/hadoop-mapreduce/lib/ > >>>>>>> snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ > >>>>>>> jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ > >>>>>>> paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- > >>>>>>> guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. > >>>>>>> jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ > >>>>>>> lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- > >>>>>>> mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ > >>>>>>> guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. > >>>>>>> 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ > >>>>>>> usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ > >>>>>>> hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- > >>>>>>> mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ > >>>>>>> hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- > >>>> yarn-server- > >>>>>>> sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > >>>>>>> server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > >>>>>>> hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > >>>>>>> hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- > >>>>>>> yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ > >>>>>>> hadoop-yarn/hadoop-yarn-applications-distributedshell- > >>>>>>> 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > >>>>>>> server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- > >>> yarn-server- > >>>>>>> sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > >>>>>>> hadoop-yarn-server-applicationhistoryservice.jar: > >>>>>>> /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ > >>>>>>> lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- > >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- > >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > >>>>>>> applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- > >>>>>>> yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ > >>>>>>> lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- > >>>>>>> yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- > >>>>>>> yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. > >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > >>>>>>> unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- > >>>>>>> yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ > >>>>>>> hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- > >>>>>>> common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- > >>>>>>> proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- > >>>>>>> nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. > >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- > >>>>>>> unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- > >>>>>>> yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- > >>>>>>> api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. > >>>>>>> jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ > >>>>>>> lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ > >>>>>>> hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ > >>>>>>> lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. > >>>>>>> 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ > >>>>>>> usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ > >>>>>>> hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ > >>>>>>> lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- > >>>>>>> client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. > >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ > >>>>>>> usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- > >>>>>>> yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ > >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ > >>>>>>> aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ > >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ > >>>>>>> usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: > >>>>>>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ > >>>>>>> lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ > >>>>>>> hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- > >>>>>>> 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. > >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ > >>>>>>> hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ > >>>>>>> lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- > >>>>>>> collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- > >>>>>>> 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ > >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ > >>>>>>> hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ > >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- > >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- > >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: > >>>>>>> /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ > >>>>>>> share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ > >>>>>>> jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ > >>>>>>> bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- > >>>>>>> over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- > >>>>>>> java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. > >>>>>>> 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ > >>>>>>> usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ > >>>>>>> share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ > >>>>>>> share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ > >>>>>>> usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ > >>>>>>> lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ > >>>>>>> share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ > >>>>>>> aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ > >>>>>>> aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ > >>>>>>> usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- > >>>>>>> gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > >>>>>>> java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>> aws-java-sdk- > >>>>>>> elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk- > >> mechanicalturkrequester-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > >>>>>>> java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- > >>>>>>> sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- > >>>>>>> sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>> stepfunctions-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > >>>>>>> java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- > >>>>>> sdk/aws-java-sdk- > >>>>>>> cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>> elastictranscoder-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>> organizations-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- > >>>>>> sdk/aws-java-sdk- > >>>>>>> applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ > >>>>>>> aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: > >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > >>>>>>> java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: > >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > >>>>>>> java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk- > >> resourcegroupstaggingapi-1.11. > >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- > >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- > >>>>>>> java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ > >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: > >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. > >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. > >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- > >>>>>>> java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ > >>>>>>> aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ > >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- > >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- > >>>>>>> elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ > >>>>>>> aws-java-sdk-iot-1.11.160.jar > >>>>>>> > >>>>>>> Thank you > >>>>>>> > >>>>>>> On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < > >>> [hidden email]> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi Andy, > >>>>>>>> > >>>>>>>> the CliFrontend is not executed via Yarn, thus, it is not > >>> affected > >>>> by > >>>>>>>> dependencies which are added due to the underlying Yarn > >> cluster. > >>>>>>> Therefore, > >>>>>>>> it would be helpful to look at the TaskManager logs. Either you > >>>> have > >>>>>>>> enabled log aggregation on your Yarn cluster, then you can > >> obtain > >>>> the > >>>>>>> logs > >>>>>>>> via `yarn logs -applicationId <application ID>` or you have to > >>>>> retrieve > >>>>>>>> them from the machines where they were running (either by going > >>>>>> directly > >>>>>>>> there or via the Yarn web interface). > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Till > >>>>>>>> > >>>>>>>> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> > >>> wrote: > >>>>>>>> > >>>>>>>>> Hi Till, > >>>>>>>>> > >>>>>>>>> That is actually the classpath used by the flink bash > >>> script(that > >>>>>>>> launches > >>>>>>>>> the jar using the java command). I changed the execute to an > >>>> echo, > >>>>>> and > >>>>>>>>> grabbed that for the CLI arguments. > >>>>>>>>> > >>>>>>>>> I believe this is the class path from the log file(although > >> it > >>>>> might > >>>>>>> not > >>>>>>>> be > >>>>>>>>> the taskmanager log, is that any different from what would be > >>> in > >>>> my > >>>>>>>>> flink-1.3.2/log folder?): > >>>>>>>>> > >>>>>>>>> 2017-10-02 20:03:26,450 INFO org.apache.flink.client. > >>>> CliFrontend > >>>>>>>>> - Classpath: > >>>>>>>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ > >>>>>>>>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. > >>>>>>>>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ > >>>>>>>>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ > >>>>>>>>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ > >>>>> hadoop/conf: > >>>>>>>>> > >>>>>>>>> If that doesn't seem right, and you can point me in the right > >>>>>> direction > >>>>>>>> as > >>>>>>>>> to where the TaskManager logs would be, I would be happy to > >>> grab > >>>>> the > >>>>>>>>> information your looking for. > >>>>>>>>> > >>>>>>>>> Thank you > >>>>>>>>> > >>>>>>>>> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < > >>>>> [hidden email]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Andy, > >>>>>>>>>> > >>>>>>>>>> this looks to me indeed like a dependency problem. I assume > >>>> that > >>>>>> EMR > >>>>>>> or > >>>>>>>>>> something else is pulling in an incompatible version of > >>> Hadoop. > >>>>>>>>>> > >>>>>>>>>> The classpath you've posted, is this the one logged in the > >>> log > >>>>>> files > >>>>>>>>>> (TaskManager log) or did you compile it yourself? In the > >>> latter > >>>>>> case, > >>>>>>>> it > >>>>>>>>>> would also be helpful to get access to the TaskManager > >> logs. > >>>>>>>>>> > >>>>>>>>>> Cheers, > >>>>>>>>>> Till > >>>>>>>>>> > >>>>>>>>>> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. < > >> [hidden email]> > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Fabian, > >>>>>>>>>>> > >>>>>>>>>>> 1) I have looked at the linked docs, and from what I can > >>> tell > >>>>> no > >>>>>>>> setup > >>>>>>>>>>> should really need to be done to get Flink working(Other > >>> than > >>>>>>>>> downloading > >>>>>>>>>>> the correct binaries, which I believe I did) > >>>>>>>>>>> 2) I have downloaded the Flink 1.3.2 > >>>> binaries(flink-1.3.2-bin- > >>>>>>>>>>> hadoop27-scala_2.11.tgz > >>>>>>>>>>> <http://apache.claz.org/flink/ > >> flink-1.3.2/flink-1.3.2-bin- > >>>>>>>>>>> hadoop27-scala_2.11.tgz>) > >>>>>>>>>>> This is for hadoop 2.7.X, which matches EMR 5.8.0. > >>>>>>>>>>> > >>>>>>>>>>> I appreciate any help or guidance you can provide me in > >>>> fixing > >>>>> my > >>>>>>>>>> problems, > >>>>>>>>>>> please let me know if there is anything else I can > >> provide > >>>> you. > >>>>>>>>>>> > >>>>>>>>>>> Thank you > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < > >>>>> [hidden email] > >>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Andy, > >>>>>>>>>>>> > >>>>>>>>>>>> I'm not an AWS expert, so I'll just check on some > >> common > >>>>>> issues. > >>>>>>>>>>>> > >>>>>>>>>>>> I guess you already had a look at the Flink docs for > >>>> AWS/EMR > >>>>>> but > >>>>>>>> I'll > >>>>>>>>>>> post > >>>>>>>>>>>> the link just be to sure [1]. > >>>>>>>>>>>> > >>>>>>>>>>>> Since you are using Flink 1.3.2 (EMR 5.8.0 comes with > >>> Flink > >>>>>>> 1.3.1) > >>>>>>>>> did > >>>>>>>>>>> you > >>>>>>>>>>>> built Flink yourself or did you download the binaries? > >>>>>>>>>>>> Does the Hadoop version of the Flink build match the > >>> Hadoop > >>>>>>> version > >>>>>>>>> of > >>>>>>>>>>> EMR > >>>>>>>>>>>> 5.8.0, i.e., Hadoop 2.7.x? > >>>>>>>>>>>> > >>>>>>>>>>>> Best, Fabian > >>>>>>>>>>>> > >>>>>>>>>>>> [1] > >>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs- > >>>>>>>>>>> release-1.3/setup/aws.html > >>>>>>>>>>>> > >>>>>>>>>>>> 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email] > >>> : > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Fabian, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Sorry, I just realized I forgot to include that part. > >>>> The > >>>>>>> error > >>>>>>>>>>> returned > >>>>>>>>>>>>> is: > >>>>>>>>>>>>> > >>>>>>>>>>>>> java.lang.NoSuchMethodError: > >>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.addResource( > >>>>>>>>>>>> Lorg/apache/hadoop/conf/ > >>>>>>>>>>>>> Configuration;)V > >>>>>>>>>>>>> at com.amazon.ws.emr.hadoop.fs. > >>>>> EmrFileSystem.initialize( > >>>>>>>>>>>>> EmrFileSystem.java:93) > >>>>>>>>>>>>> at org.apache.flink.runtime.fs. > >>>> hdfs.HadoopFileSystem. > >>>>>>>>>>>>> initialize(HadoopFileSystem.java:328) > >>>>>>>>>>>>> at org.apache.flink.core.fs.FileSystem. > >>>>>>>> getUnguardedFileSystem( > >>>>>>>>>>>>> FileSystem.java:350) > >>>>>>>>>>>>> at org.apache.flink.core.fs. > >>>> FileSystem.get(FileSystem. > >>>>>>>>> java:389) > >>>>>>>>>>>>> at org.apache.flink.core.fs.Path. > >>>>>>>> getFileSystem(Path.java:293) > >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. > >>>>>>>>>>>>> FsCheckpointStreamFactory.<init>( > >>>>> FsCheckpointStreamFactory. > >>>>>>>>> java:99) > >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. > >>>>>>> FsStateBackend. > >>>>>>>>>>>>> createStreamFactory(FsStateBackend.java:282) > >>>>>>>>>>>>> at org.apache.flink.contrib.streaming.state. > >>>>>>>>> RocksDBStateBackend. > >>>>>>>>>>>>> createStreamFactory(RocksDBStateBackend.java:273 > >>>>>>>>>>>>> > >>>>>>>>>>>>> I believe it has something to do with the classpath, > >>> but > >>>> I > >>>>> am > >>>>>>>>> unsure > >>>>>>>>>>> why > >>>>>>>>>>>> or > >>>>>>>>>>>>> how to fix it. The classpath being used during the > >>>>> execution > >>>>>>> is: > >>>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/ > >>>> flink-python_2.11-1.3.2.jar:/ > >>>>>>>>>>>>> home/hadoop/flink-1.3.2/ > >>> lib/flink-shaded-hadoop2- > >>>>>>>>>>>>> uber-1.3.2.jar:/home/hadoop/ > >>>>> flink-1.3.2/lib/log4j-1.2. > >>>>>>>>>>>>> 17.jar:/home/hadoop/flink-1. > >>>>> 3.2/lib/slf4j-log4j12-1.7.7. > >>>>>>>>>>>>> jar:/home/hadoop/flink-1. > >>>>> 3.2/lib/flink-dist_2.11-1.3. > >>>>>>>>>>>>> 2.jar::/etc/hadoop/conf: > >>>>>>>>>>>>> > >>>>>>>>>>>>> I decompiled flink-shaded-hadoop2-uber-1.3.2.jar > >>> and > >>>>> it > >>>>>>>> seems > >>>>>>>>>> the > >>>>>>>>>>>>> addResource function does seem to be there. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thank you > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < > >>>>>>> [hidden email] > >>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Andy, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> can you describe in more detail what exactly isn't > >>>>> working? > >>>>>>>>>>>>>> Do you see error messages in the log files or on > >> the > >>>>>> console? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, Fabian > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2017-10-02 15:52 GMT+02:00 Andy M. < > >>> [hidden email] > >>>>> : > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I am about to deploy my first Flink projects to > >>>>>>> production, > >>>>>>>>> but > >>>>>>>>>> I > >>>>>>>>>>> am > >>>>>>>>>>>>>>> running into a very big hurdle. I am unable to > >>>> launch > >>>>> my > >>>>>>>>> project > >>>>>>>>>>> so > >>>>>>>>>>>> it > >>>>>>>>>>>>>> can > >>>>>>>>>>>>>>> write to an S3 bucket. My project is running on > >> an > >>>> EMR > >>>>>>>>> cluster, > >>>>>>>>>>>> where > >>>>>>>>>>>>> I > >>>>>>>>>>>>>>> have installed Flink 1.3.2. I am using Yarn to > >>>> launch > >>>>>> the > >>>>>>>>>>>> application, > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>> it seems to run fine unless I am trying to enable > >>>> check > >>>>>>>>>>>> pointing(with a > >>>>>>>>>>>>>> S3 > >>>>>>>>>>>>>>> target). I am looking to use RocksDB as my > >>>>>> check-pointing > >>>>>>>>>> backend. > >>>>>>>>>>>> I > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>>> asked a few places, and I am still unable to > >> find a > >>>>>>> solution > >>>>>>>> to > >>>>>>>>>>> this > >>>>>>>>>>>>>>> problem. Here are my steps for creating a > >> cluster, > >>>> and > >>>>>>>>> launching > >>>>>>>>>>> my > >>>>>>>>>>>>>>> application, perhaps I am missing a step. I'd be > >>>> happy > >>>>>> to > >>>>>>>>>> provide > >>>>>>>>>>>> any > >>>>>>>>>>>>>>> additional information if needed. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> AWS Portal: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 1) EMR -> Create Cluster > >>>>>>>>>>>>>>> 2) Advanced Options > >>>>>>>>>>>>>>> 3) Release = emr-5.8.0 > >>>>>>>>>>>>>>> 4) Only select Hadoop 2.7.3 > >>>>>>>>>>>>>>> 5) Next -> Next -> Next -> Create Cluster ( I > >>> do > >>>>> fill > >>>>>>> out > >>>>>>>>>>>>>>> names/keys/etc) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Once the cluster is up I ssh into the Master and > >> do > >>>> the > >>>>>>>>>> following: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 1 wget > >>>>>>>>>>>>>>> http://apache.claz.org/flink/ > >>>>>> flink-1.3.2/flink-1.3.2-bin- > >>>>>>>>>>>>>>> hadoop27-scala_2.11.tgz > >>>>>>>>>>>>>>> 2 tar -xzf flink-1.3.2-bin-hadoop27- > >>>>> scala_2.11.tgz > >>>>>>>>>>>>>>> 3 cd flink-1.3.2 > >>>>>>>>>>>>>>> 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 > >> -d > >>>>>>>>>>>>>>> 5 Change conf/flink-conf.yaml > >>>>>>>>>>>>>>> 6 ./bin/flink run -m yarn-cluster -yn 1 > >>>>>>>>> ~/flink-consumer.jar > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> My conf/flink-conf.yaml I add the following > >> fields: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> state.backend: rocksdb > >>>>>>>>>>>>>>> state.backend.fs.checkpointdir: > >>>>> s3:/bucket/location > >>>>>>>>>>>>>>> state.checkpoints.dir: s3:/bucket/location > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> My program's checkpointing setup: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> env.enableCheckpointing(getCheckpointRate, > >>>>>>>>>>> CheckpointingMode.EXACTLY_ > >>>>>>>>>>>>>> ONCE) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> env.getCheckpointConfig. > >>>> enableExternalizedCheckpoints( > >>>>>>>>>>>>>>> ExternalizedCheckpointCleanup. > >>>> RETAIN_ON_CANCELLATION) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> env.getCheckpointConfig. > >>>> setMinPauseBetweenCheckpoints( > >>>>>>>>>>>>>>> getCheckpointMinPause) > >>>>>>>>>>>>>>> env.getCheckpointConfig. > >> setCheckpointTimeout( > >>>>>>>>>>>> getCheckpointTimeout) > >>>>>>>>>>>>>>> env.getCheckpointConfig. > >>>>>> setMaxConcurrentCheckpoints(1) > >>>>>>>>>>>>>>> env.setStateBackend(new > >>>> RocksDBStateBackend("s3:// > >>>>>>>>>>>>> bucket/location", > >>>>>>>>>>>>>>> true)) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > > |
Hi,
I can't spot anything obviously wrong with your config. I've been quite busy lately with traveling and release preparations. I will get back to this and try it myself, though, once I have time again. Best, Aljoscha > On 12. Oct 2017, at 17:07, Andy M. <[hidden email]> wrote: > > Hi Ajoscha, > > That didn't seem to do the trick either. Do the following look correct? > > I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same > problem, all I did was upload my Scala .jar to the master, updated my > flink-conf.yaml, set my env variables, and ran it with the following > command: HADOOP_CONF_DIR=/etc/hadoop/conf flink run -m yarn-cluster -yn 5 > -ys 2 ~/flink-consumer.jar > > I am beginning to think something else may be wrong with my configuration. > Does the following look correct? > > In flink-conf.yaml: > state.backend: rocksdb > state.backend.fs.checkpointdir: s3://org/flink-project/state > state.checkpoints.dir: s3://org/flink-project/state > > In the code: > env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE) > env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) > env.getCheckpointConfig.setMinPauseBetweenCheckpoints(1000) > env.getCheckpointConfig.setCheckpointTimeout(5000) > env.getCheckpointConfig.setMaxConcurrentCheckpoints(1) > env.setStateBackend(new RocksDBStateBackend("s3://org/flink-project/state", > true)) > > Thank you > > > > On Wed, Oct 11, 2017 at 5:19 AM, Aljoscha Krettek <[hidden email]> > wrote: > >> Hi Andy, >> >> I remember that I was testing a job with almost exactly the same setup as >> part of the Flink 1.3.2 release testing. The command I used to start my job >> is roughly this: >> >> HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m >> yarn-cluster -yn 5 -ys 2 ... >> >> i.e. I export the proper hadoop config dir and I run a per-job YARN >> cluster. I think I also exported the result of "hadoop classpath" as >> HADOOP_CLASSPATH. >> >> Best, >> Aljoscha >> >>> On 10. Oct 2017, at 16:43, Andy M. <[hidden email]> wrote: >>> >>> Hello, >>> >>> Bowen: Unless I am missing something, it says there needs to be no setup >>> on EMR, Each topic says: "You don’t have to configure this manually if >> you >>> are running Flink on EMR." S3 access from CLI works fine on my clusters. >>> >>> Chen: Thank you for this, I will look into this if I am unable to get >> this >>> running on YARN successfully. >>> >>> Stephan: Removing the said library causes the flink >>> (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs >>> this to work. I tried explicitly setting the classpath for the java call >>> as well to point to the hadoop library jars. This is the original java >>> command that I was trying to run: >>> >>> java >>> -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client- >> ip-172-31-19-27.log >>> -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/ >> log4j-cli.properties >>> -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/ >> conf/logback.xml >>> -classpath >>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ >> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. >> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ >> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ >> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: >>> org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 >>> /home/hadoop/flink-consumer.jar >>> >>> >>> This is what I changed it too(removing the shadded-hadoop2-uber jar and >>> adding in the hadoop folder): >>> >>> java >>> -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client- >> ip-172-31-19-27.log >>> -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/ >> log4j-cli.properties >>> -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/ >> conf/logback.xml >>> -classpath >>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ >> home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/ >> flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink- >> 1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/ >> activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar: >> /usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/ >> lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/ >> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- >> lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/ >> usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/ >> lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/ >> lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java- >> xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final. >> jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/ >> lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/ >> jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/ >> usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ >> lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2. >> 3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ >> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator- >> client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9. >> jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/ >> hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator- >> framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9. >> jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/ >> hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/ >> curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey- >> server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7. >> 10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. >> jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/ >> jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. >> 1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/ >> hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1. >> 1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/ >> hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ >> hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26- >> emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons- >> collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3. >> 1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26- >> emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/ >> lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/ >> httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/ >> usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/ >> lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/ >> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/ >> usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/ >> hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/ >> jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient- >> 3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/ >> lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf >>> org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 >>> /home/hadoop/flink-consumer.jar >>> >>> The later throws the following error: >>> >>> SLF4J: Class path contains multiple SLF4J bindings. >>> SLF4J: Found binding in >>> [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7. >> jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: Found binding in >>> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/ >> slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>> explanation. >>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/hadoop/util/VersionInfo >>> at >>> org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo( >> EnvironmentInformation.java:283) >>> at org.apache.flink.client.CliFrontend.main(CliFrontend. >> java:1124) >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.hadoop.util.VersionInfo >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> ... 2 more >>> >>> Simply removing the .jar from the folder causes the same error. >>> >>> Thank you >>> >>> >>> >>> >>> On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <[hidden email]> wrote: >>> >>>> Hi! >>>> >>>> It looks like multiple Hadoop versions are in the classpath. Flink's >> hadoop >>>> jar and the EMR Hadoop jars. >>>> I would simply drop Flink's own Hadoop dependency and only use the EMR >>>> Hadoop jars. >>>> >>>> Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure >> the >>>> setup is such that the Hadoop lib environment variable is set. Then it >>>> should not have conflicts any more. >>>> >>>> >>>> >>>> On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <[hidden email]> wrote: >>>> >>>>> Attached my side project verified working to deploy jobmanager and >>>>> taskmanager as stateless service(non yarn/mesos), configuration here >>>>> >>>>> https://github.com/chenqin/flink-jar/tree/master/config/hadoop >>>>> >>>>> more detail here >>>>> https://github.com/chenqin/flink-jar/blob/master/src/ >>>>> main/java/FlinkBootstrap.java#L49 >>>>> >>>>> On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <[hidden email]> >>>> wrote: >>>>> >>>>>> Hi Andy, >>>>>> >>>>>> I believe it's because you didn't set your s3 impl correctly. Try to >>>> set >>>>>> your core-site.xml by following https://ci.apache.org/ >>>>>> projects/flink/flink-docs-release-1.4/ops/deployment/ >>>>>> aws.html#s3afilesystem- >>>>>> recommended >>>>>> >>>>>> Bowen >>>>>> >>>>>> On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <[hidden email]> wrote: >>>>>> >>>>>>> Hi Till, >>>>>>> >>>>>>> Seems like everything is in line there. hadoop-common.jar -> >>>>>>> hadoop-common-2.7.3-amzn-3.jar >>>>>>> >>>>>>> And when i decompiled that jar I see public void >>>>>> addResource(Configuration >>>>>>> conf) in org/apache/hadoop/conf/Configuration.java >>>>>>> >>>>>>> I agree that an incorrect version of the jar is probably being run, >>>> is >>>>>>> there a way to limit the classpath for the TaskManager when starting >>>>> the >>>>>>> job? >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <[hidden email]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Andy, >>>>>>>> >>>>>>>> could you check which Hadoop version this jar >>>>>>>> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether >>>>> the >>>>>>>> contained hadoop Configuration class has the method >>>>>>>> Configuration.addResource(Lorg/apache/hadoop/conf/ >>>> Configuration;)V. >>>>>>> Maybe >>>>>>>> this jar is the culprit because it comes from a different Hadoop >>>>>> version. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <[hidden email]> wrote: >>>>>>>> >>>>>>>>> Hi Till, >>>>>>>>> >>>>>>>>> I believe this is what you are looking for, classpath is much >>>>> bigger >>>>>>> for >>>>>>>>> the task manager. I can also post the whole log file if needed: >>>>>>>>> >>>>>>>>> 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. >>>>>>>> YarnTaskManagerRunner >>>>>>>>> - Classpath: >>>>>>>>> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- >>>>>>>>> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. >>>>>>>>> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: >>>>>>>>> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ >>>>>>>>> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- >>>>>>>>> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. >>>>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>>>> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- >>>>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- >>>>>>>>> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ >>>>>>>>> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. >>>>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ >>>>>>>>> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. >>>>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- >>>>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. >>>>>>>>> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ >>>>>>>>> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>>>> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>>>> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ >>>>>>>>> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- >>>>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. >>>>>>>>> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. >>>>>>>>> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ >>>>>>>>> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ >>>>>>>>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. >>>>>>>>> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. >>>>>>>>> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ >>>>>>>>> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- >>>>>>>>> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ >>>>>>>>> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ >>>>>>>>> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. >>>>>>>>> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ >>>>>>>>> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ >>>>>>>>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- >>>>>>>>> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. >>>>>>>>> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ >>>>>>>>> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ >>>>>>>>> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ >>>>>>>>> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ >>>>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. >>>>>>>>> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ >>>>>>>>> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. >>>>>>>>> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ >>>>>>>>> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ >>>>>>>>> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. >>>>>>>>> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ >>>>>>>>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- >>>>>>>>> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- >>>>>>>>> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: >>>>>>>>> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ >>>>>>>>> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- >>>>>>>>> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ >>>>>>>>> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ >>>>>>>>> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ >>>>>>>>> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. >>>>>>>>> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: >>>>>>>>> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ >>>>>>>>> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. >>>>>>>>> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ >>>>>>>>> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ >>>>>>>>> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. >>>>>>>>> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ >>>>>>>>> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ >>>>>>>>> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- >>>>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. >>>>>>>>> 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ >>>>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ >>>>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ >>>>>>>>> hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- >>>>>>>>> amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- >>>>>>>>> incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- >>>>>>>>> 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. >>>>>>>>> jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ >>>>>>>>> lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ >>>>>>>>> lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ >>>>>>>>> commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. >>>>>>>>> jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ >>>>>>>>> usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ >>>>>>>>> hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ >>>>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- >>>>>>>>> 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ >>>>>>>>> hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ >>>>>>>>> lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ >>>>>>>>> hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- >>>>>>>>> hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ >>>>>>>>> lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- >>>>>>>>> api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: >>>>>>>>> /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ >>>>>>>>> hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ >>>>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- >>>>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ >>>>>>>> hadoop-mapreduce/hadoop- >>>>>>>>> mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- >>>>>>>>> mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- >>>>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. >>>>>>>> 8.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- >>>>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- >>>>>>>>> net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- >>>>>>>>> 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- >>>>>>>>> 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: >>>>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. >>>>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. >>>>>>>>> 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- >>>>>>>>> client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- >>>>>>>>> common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. >>>>>>>>> 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- >>>>>>>>> jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- >>>>>>>>> archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. >>>>>>>>> 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- >>>>>>>>> kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: >>>>>>>>> /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: >>>>>>>>> /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- >>>>>> mapreduce/commons- >>>>>>>>> configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- >>>>>>>>> 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- >>>>>>>>> 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: >>>>>>>>> /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ >>>>>>>>> lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ >>>>>>>> lib/hadoop-mapreduce/lib/ >>>>>>>>> snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>>>> jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>>>> paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- >>>>>>>>> guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. >>>>>>>>> jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>>>> lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- >>>>>>>>> mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>>>> guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. >>>>>>>>> 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ >>>>>>>>> usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ >>>>>>>>> hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- >>>>>>>>> mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- >>>>>> yarn-server- >>>>>>>>> sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>>>> server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>>>> yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ >>>>>>>>> hadoop-yarn/hadoop-yarn-applications-distributedshell- >>>>>>>>> 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>>>> server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- >>>>> yarn-server- >>>>>>>>> sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> hadoop-yarn-server-applicationhistoryservice.jar: >>>>>>>>> /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ >>>>>>>>> lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- >>>>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- >>>>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- >>>>>>>>> applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>>>> yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ >>>>>>>>> lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- >>>>>>>>> yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>>>> yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. >>>>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- >>>>>>>>> unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- >>>>>>>>> yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>>>> common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- >>>>>>>>> proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- >>>>>>>>> nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. >>>>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- >>>>>>>>> unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>>>> yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- >>>>>>>>> api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. >>>>>>>>> jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ >>>>>>>>> lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ >>>>>>>>> hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. >>>>>>>>> 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ >>>>>>>>> usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ >>>>>>>>> hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- >>>>>>>>> client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. >>>>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ >>>>>>>>> usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- >>>>>>>>> yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ >>>>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ >>>>>>>>> aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ >>>>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ >>>>>>>>> usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: >>>>>>>>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ >>>>>>>>> lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ >>>>>>>>> hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- >>>>>>>>> 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. >>>>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ >>>>>>>>> hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- >>>>>>>>> collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- >>>>>>>>> 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ >>>>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ >>>>>>>>> hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ >>>>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- >>>>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- >>>>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: >>>>>>>>> /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ >>>>>>>>> share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ >>>>>>>>> jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ >>>>>>>>> bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- >>>>>>>>> over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- >>>>>>>>> java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. >>>>>>>>> 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ >>>>>>>>> usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ >>>>>>>>> share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ >>>>>>>>> share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ >>>>>>>>> usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ >>>>>>>>> lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ >>>>>>>>> share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ >>>>>>>>> aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ >>>>>>>>> aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ >>>>>>>>> usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- >>>>>>>>> gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>>>> java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk- >>>>>>>>> elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk- >>>> mechanicalturkrequester-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>>>> java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>>>> sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>>>> sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>> stepfunctions-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>>>> java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>>> sdk/aws-java-sdk- >>>>>>>>> cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elastictranscoder-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>> organizations-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>>> sdk/aws-java-sdk- >>>>>>>>> applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ >>>>>>>>> aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: >>>>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>>>> java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: >>>>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>>>> java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk- >>>> resourcegroupstaggingapi-1.11. >>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- >>>>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>>>> java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ >>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: >>>>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. >>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. >>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- >>>>>>>>> java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ >>>>>>>>> aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ >>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- >>>>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>>>> elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>>>> aws-java-sdk-iot-1.11.160.jar >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < >>>>> [hidden email]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Andy, >>>>>>>>>> >>>>>>>>>> the CliFrontend is not executed via Yarn, thus, it is not >>>>> affected >>>>>> by >>>>>>>>>> dependencies which are added due to the underlying Yarn >>>> cluster. >>>>>>>>> Therefore, >>>>>>>>>> it would be helpful to look at the TaskManager logs. Either you >>>>>> have >>>>>>>>>> enabled log aggregation on your Yarn cluster, then you can >>>> obtain >>>>>> the >>>>>>>>> logs >>>>>>>>>> via `yarn logs -applicationId <application ID>` or you have to >>>>>>> retrieve >>>>>>>>>> them from the machines where they were running (either by going >>>>>>>> directly >>>>>>>>>> there or via the Yarn web interface). >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Till >>>>>>>>>> >>>>>>>>>> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <[hidden email]> >>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Till, >>>>>>>>>>> >>>>>>>>>>> That is actually the classpath used by the flink bash >>>>> script(that >>>>>>>>>> launches >>>>>>>>>>> the jar using the java command). I changed the execute to an >>>>>> echo, >>>>>>>> and >>>>>>>>>>> grabbed that for the CLI arguments. >>>>>>>>>>> >>>>>>>>>>> I believe this is the class path from the log file(although >>>> it >>>>>>> might >>>>>>>>> not >>>>>>>>>> be >>>>>>>>>>> the taskmanager log, is that any different from what would be >>>>> in >>>>>> my >>>>>>>>>>> flink-1.3.2/log folder?): >>>>>>>>>>> >>>>>>>>>>> 2017-10-02 20:03:26,450 INFO org.apache.flink.client. >>>>>> CliFrontend >>>>>>>>>>> - Classpath: >>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ >>>>>>>>>>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. >>>>>>>>>>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ >>>>>>>>>>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ >>>>>>>>>>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ >>>>>>> hadoop/conf: >>>>>>>>>>> >>>>>>>>>>> If that doesn't seem right, and you can point me in the right >>>>>>>> direction >>>>>>>>>> as >>>>>>>>>>> to where the TaskManager logs would be, I would be happy to >>>>> grab >>>>>>> the >>>>>>>>>>> information your looking for. >>>>>>>>>>> >>>>>>>>>>> Thank you >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < >>>>>>> [hidden email]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Andy, >>>>>>>>>>>> >>>>>>>>>>>> this looks to me indeed like a dependency problem. I assume >>>>>> that >>>>>>>> EMR >>>>>>>>> or >>>>>>>>>>>> something else is pulling in an incompatible version of >>>>> Hadoop. >>>>>>>>>>>> >>>>>>>>>>>> The classpath you've posted, is this the one logged in the >>>>> log >>>>>>>> files >>>>>>>>>>>> (TaskManager log) or did you compile it yourself? In the >>>>> latter >>>>>>>> case, >>>>>>>>>> it >>>>>>>>>>>> would also be helpful to get access to the TaskManager >>>> logs. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Till >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. < >>>> [hidden email]> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Fabian, >>>>>>>>>>>>> >>>>>>>>>>>>> 1) I have looked at the linked docs, and from what I can >>>>> tell >>>>>>> no >>>>>>>>>> setup >>>>>>>>>>>>> should really need to be done to get Flink working(Other >>>>> than >>>>>>>>>>> downloading >>>>>>>>>>>>> the correct binaries, which I believe I did) >>>>>>>>>>>>> 2) I have downloaded the Flink 1.3.2 >>>>>> binaries(flink-1.3.2-bin- >>>>>>>>>>>>> hadoop27-scala_2.11.tgz >>>>>>>>>>>>> <http://apache.claz.org/flink/ >>>> flink-1.3.2/flink-1.3.2-bin- >>>>>>>>>>>>> hadoop27-scala_2.11.tgz>) >>>>>>>>>>>>> This is for hadoop 2.7.X, which matches EMR 5.8.0. >>>>>>>>>>>>> >>>>>>>>>>>>> I appreciate any help or guidance you can provide me in >>>>>> fixing >>>>>>> my >>>>>>>>>>>> problems, >>>>>>>>>>>>> please let me know if there is anything else I can >>>> provide >>>>>> you. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < >>>>>>> [hidden email] >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Andy, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not an AWS expert, so I'll just check on some >>>> common >>>>>>>> issues. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess you already had a look at the Flink docs for >>>>>> AWS/EMR >>>>>>>> but >>>>>>>>>> I'll >>>>>>>>>>>>> post >>>>>>>>>>>>>> the link just be to sure [1]. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Since you are using Flink 1.3.2 (EMR 5.8.0 comes with >>>>> Flink >>>>>>>>> 1.3.1) >>>>>>>>>>> did >>>>>>>>>>>>> you >>>>>>>>>>>>>> built Flink yourself or did you download the binaries? >>>>>>>>>>>>>> Does the Hadoop version of the Flink build match the >>>>> Hadoop >>>>>>>>> version >>>>>>>>>>> of >>>>>>>>>>>>> EMR >>>>>>>>>>>>>> 5.8.0, i.e., Hadoop 2.7.x? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, Fabian >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs- >>>>>>>>>>>>> release-1.3/setup/aws.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-10-02 21:51 GMT+02:00 Andy M. <[hidden email] >>>>> : >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Fabian, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sorry, I just realized I forgot to include that part. >>>>>> The >>>>>>>>> error >>>>>>>>>>>>> returned >>>>>>>>>>>>>>> is: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> java.lang.NoSuchMethodError: >>>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.addResource( >>>>>>>>>>>>>> Lorg/apache/hadoop/conf/ >>>>>>>>>>>>>>> Configuration;)V >>>>>>>>>>>>>>> at com.amazon.ws.emr.hadoop.fs. >>>>>>> EmrFileSystem.initialize( >>>>>>>>>>>>>>> EmrFileSystem.java:93) >>>>>>>>>>>>>>> at org.apache.flink.runtime.fs. >>>>>> hdfs.HadoopFileSystem. >>>>>>>>>>>>>>> initialize(HadoopFileSystem.java:328) >>>>>>>>>>>>>>> at org.apache.flink.core.fs.FileSystem. >>>>>>>>>> getUnguardedFileSystem( >>>>>>>>>>>>>>> FileSystem.java:350) >>>>>>>>>>>>>>> at org.apache.flink.core.fs. >>>>>> FileSystem.get(FileSystem. >>>>>>>>>>> java:389) >>>>>>>>>>>>>>> at org.apache.flink.core.fs.Path. >>>>>>>>>> getFileSystem(Path.java:293) >>>>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. >>>>>>>>>>>>>>> FsCheckpointStreamFactory.<init>( >>>>>>> FsCheckpointStreamFactory. >>>>>>>>>>> java:99) >>>>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. >>>>>>>>> FsStateBackend. >>>>>>>>>>>>>>> createStreamFactory(FsStateBackend.java:282) >>>>>>>>>>>>>>> at org.apache.flink.contrib.streaming.state. >>>>>>>>>>> RocksDBStateBackend. >>>>>>>>>>>>>>> createStreamFactory(RocksDBStateBackend.java:273 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I believe it has something to do with the classpath, >>>>> but >>>>>> I >>>>>>> am >>>>>>>>>>> unsure >>>>>>>>>>>>> why >>>>>>>>>>>>>> or >>>>>>>>>>>>>>> how to fix it. The classpath being used during the >>>>>>> execution >>>>>>>>> is: >>>>>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/ >>>>>> flink-python_2.11-1.3.2.jar:/ >>>>>>>>>>>>>>> home/hadoop/flink-1.3.2/ >>>>> lib/flink-shaded-hadoop2- >>>>>>>>>>>>>>> uber-1.3.2.jar:/home/hadoop/ >>>>>>> flink-1.3.2/lib/log4j-1.2. >>>>>>>>>>>>>>> 17.jar:/home/hadoop/flink-1. >>>>>>> 3.2/lib/slf4j-log4j12-1.7.7. >>>>>>>>>>>>>>> jar:/home/hadoop/flink-1. >>>>>>> 3.2/lib/flink-dist_2.11-1.3. >>>>>>>>>>>>>>> 2.jar::/etc/hadoop/conf: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I decompiled flink-shaded-hadoop2-uber-1.3.2.jar >>>>> and >>>>>>> it >>>>>>>>>> seems >>>>>>>>>>>> the >>>>>>>>>>>>>>> addResource function does seem to be there. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < >>>>>>>>> [hidden email] >>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Andy, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> can you describe in more detail what exactly isn't >>>>>>> working? >>>>>>>>>>>>>>>> Do you see error messages in the log files or on >>>> the >>>>>>>> console? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, Fabian >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2017-10-02 15:52 GMT+02:00 Andy M. < >>>>> [hidden email] >>>>>>> : >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am about to deploy my first Flink projects to >>>>>>>>> production, >>>>>>>>>>> but >>>>>>>>>>>> I >>>>>>>>>>>>> am >>>>>>>>>>>>>>>>> running into a very big hurdle. I am unable to >>>>>> launch >>>>>>> my >>>>>>>>>>> project >>>>>>>>>>>>> so >>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>> write to an S3 bucket. My project is running on >>>> an >>>>>> EMR >>>>>>>>>>> cluster, >>>>>>>>>>>>>> where >>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>> have installed Flink 1.3.2. I am using Yarn to >>>>>> launch >>>>>>>> the >>>>>>>>>>>>>> application, >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> it seems to run fine unless I am trying to enable >>>>>> check >>>>>>>>>>>>>> pointing(with a >>>>>>>>>>>>>>>> S3 >>>>>>>>>>>>>>>>> target). I am looking to use RocksDB as my >>>>>>>> check-pointing >>>>>>>>>>>> backend. >>>>>>>>>>>>>> I >>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> asked a few places, and I am still unable to >>>> find a >>>>>>>>> solution >>>>>>>>>> to >>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> problem. Here are my steps for creating a >>>> cluster, >>>>>> and >>>>>>>>>>> launching >>>>>>>>>>>>> my >>>>>>>>>>>>>>>>> application, perhaps I am missing a step. I'd be >>>>>> happy >>>>>>>> to >>>>>>>>>>>> provide >>>>>>>>>>>>>> any >>>>>>>>>>>>>>>>> additional information if needed. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> AWS Portal: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) EMR -> Create Cluster >>>>>>>>>>>>>>>>> 2) Advanced Options >>>>>>>>>>>>>>>>> 3) Release = emr-5.8.0 >>>>>>>>>>>>>>>>> 4) Only select Hadoop 2.7.3 >>>>>>>>>>>>>>>>> 5) Next -> Next -> Next -> Create Cluster ( I >>>>> do >>>>>>> fill >>>>>>>>> out >>>>>>>>>>>>>>>>> names/keys/etc) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Once the cluster is up I ssh into the Master and >>>> do >>>>>> the >>>>>>>>>>>> following: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1 wget >>>>>>>>>>>>>>>>> http://apache.claz.org/flink/ >>>>>>>> flink-1.3.2/flink-1.3.2-bin- >>>>>>>>>>>>>>>>> hadoop27-scala_2.11.tgz >>>>>>>>>>>>>>>>> 2 tar -xzf flink-1.3.2-bin-hadoop27- >>>>>>> scala_2.11.tgz >>>>>>>>>>>>>>>>> 3 cd flink-1.3.2 >>>>>>>>>>>>>>>>> 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 >>>> -d >>>>>>>>>>>>>>>>> 5 Change conf/flink-conf.yaml >>>>>>>>>>>>>>>>> 6 ./bin/flink run -m yarn-cluster -yn 1 >>>>>>>>>>> ~/flink-consumer.jar >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> My conf/flink-conf.yaml I add the following >>>> fields: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> state.backend: rocksdb >>>>>>>>>>>>>>>>> state.backend.fs.checkpointdir: >>>>>>> s3:/bucket/location >>>>>>>>>>>>>>>>> state.checkpoints.dir: s3:/bucket/location >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> My program's checkpointing setup: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> env.enableCheckpointing(getCheckpointRate, >>>>>>>>>>>>> CheckpointingMode.EXACTLY_ >>>>>>>>>>>>>>>> ONCE) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> env.getCheckpointConfig. >>>>>> enableExternalizedCheckpoints( >>>>>>>>>>>>>>>>> ExternalizedCheckpointCleanup. >>>>>> RETAIN_ON_CANCELLATION) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> env.getCheckpointConfig. >>>>>> setMinPauseBetweenCheckpoints( >>>>>>>>>>>>>>>>> getCheckpointMinPause) >>>>>>>>>>>>>>>>> env.getCheckpointConfig. >>>> setCheckpointTimeout( >>>>>>>>>>>>>> getCheckpointTimeout) >>>>>>>>>>>>>>>>> env.getCheckpointConfig. >>>>>>>> setMaxConcurrentCheckpoints(1) >>>>>>>>>>>>>>>>> env.setStateBackend(new >>>>>> RocksDBStateBackend("s3:// >>>>>>>>>>>>>>> bucket/location", >>>>>>>>>>>>>>>>> true)) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> >> |
Free forum by Nabble | Edit this page |