Hello,
I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class. However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package. Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components? mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase [ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase RegardsVijay |
Hi Vijay,
did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes? Regards, Robert On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan < [hidden email]> wrote: > Hello, > I was trying to include "hadoop-minikdc" component to Yarn test framework > by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. > > <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> > The dependency inclusion seems to be working from IDE. IntelliJ picked up > the jar and did not report any compilation issue when I create a new test > program in "flink-yarn-tests" module and try to use the minikdc class. > However, when I try to run the test using maven command, I am getting > compilation error for the MiniKdc package. > Could someone please share some insights on how the dependency inclusion > works with the shaded hadoop components? > mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests > -Dtest=YARNSessionFIFOITCase > > [ERROR] COMPILATION ERROR : [INFO] > -------------------------------------------------------------[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] > package org.apache.hadoop.minikdc does not exist[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] > cannot find symbol symbol: variable MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase > RegardsVijay |
Hi Rob,
Yes I checked but the result jar does not contain the classes from hadoop-minikdc package. RegardsVijay On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> wrote: Hi Vijay, did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes? Regards,Robert On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <[hidden email]> wrote: Hello, I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class. However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package. Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components? mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase [ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase RegardsVijay |
Hi,
I didn't even get the shading module to build: [INFO] ------------------------------------------------------------------------ [INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 0.756 s [INFO] Finished at: 2016-06-21T17:25:00+02:00 [INFO] Final Memory: 16M/303M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project flink-shaded-include-yarn-tests: Could not resolve dependencies for project org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1] [ERROR] How did you do it? Regards, Robert On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan < [hidden email]> wrote: > Hi Rob, > Yes I checked but the result jar does not contain the classes from > hadoop-minikdc package. > RegardsVijay > > On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi Vijay, > did you check if the artifact produced by the > "flink-shaded-include-yarn-tests" module contains the required classes? > Regards,Robert > > On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hello, > I was trying to include "hadoop-minikdc" component to Yarn test framework > by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. > > <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> > The dependency inclusion seems to be working from IDE. IntelliJ picked up > the jar and did not report any compilation issue when I create a new test > program in "flink-yarn-tests" module and try to use the minikdc class. > However, when I try to run the test using maven command, I am getting > compilation error for the MiniKdc package. > Could someone please share some insights on how the dependency inclusion > works with the shaded hadoop components? > mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests > -Dtest=YARNSessionFIFOITCase > > [ERROR] COMPILATION ERROR : [INFO] > -------------------------------------------------------------[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] > package org.apache.hadoop.minikdc does not exist[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] > cannot find symbol symbol: variable MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase > RegardsVijay > > > > > |
Hi Rob,
You need to include below lines to the pom.xml to resolve the chain dependency error. <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC dependency error --> <build> <plugins> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> <extensions>true</extensions> </plugin> </plugins> </build> RegardsVijay On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <[hidden email]> wrote: Hi, I didn't even get the shading module to build: [INFO] ------------------------------------------------------------------------[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] Final Memory: 16M/303M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal on project flink-shaded-include-yarn-tests: Could not resolve dependencies for project org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1][ERROR] How did you do it? Regards,Robert On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan <[hidden email]> wrote: Hi Rob, Yes I checked but the result jar does not contain the classes from hadoop-minikdc package. RegardsVijay On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> wrote: Hi Vijay, did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes? Regards,Robert On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <[hidden email]> wrote: Hello, I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class. However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package. Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components? mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase [ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase RegardsVijay |
Cool, thank you.
After adding the line you've mentioned, my jar contains the missing MiniKdc class: robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdc minikdc-krb5.conf minikdc.ldiff org/apache/hadoop/minikdc/ org/apache/hadoop/minikdc/KerberosSecurityTestcase.class org/apache/hadoop/minikdc/MiniKdc$1.class org/apache/hadoop/minikdc/MiniKdc.class META-INF/maven/org.apache.hadoop/hadoop-minikdc/ META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % Is the same true on your machine? On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan < [hidden email]> wrote: > Hi Rob, > You need to include below lines to the pom.xml to resolve the chain > dependency error. > <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC > dependency error --> <build> <plugins> <plugin> > <groupId>org.apache.felix</groupId> > <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> > <extensions>true</extensions> </plugin> </plugins> </build> > RegardsVijay > > On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi, > I didn't even get the shading module to build: > [INFO] > ------------------------------------------------------------------------[INFO] > Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] > ------------------------------------------------------------------------[INFO] > ------------------------------------------------------------------------[INFO] > BUILD FAILURE[INFO] > ------------------------------------------------------------------------[INFO] > Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] > Final Memory: 16M/303M[INFO] > ------------------------------------------------------------------------[ERROR] > Failed to execute goal on project flink-shaded-include-yarn-tests: Could > not resolve dependencies for project > org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure > to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in > https://repo.maven.apache.org/maven2 was cached in the local repository, > resolution will not be reattempted until the update interval of central has > elapsed or updates are forced -> [Help 1][ERROR] > > How did you do it? > Regards,Robert > > On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hi Rob, > Yes I checked but the result jar does not contain the classes from > hadoop-minikdc package. > RegardsVijay > > On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi Vijay, > did you check if the artifact produced by the > "flink-shaded-include-yarn-tests" module contains the required classes? > Regards,Robert > > On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hello, > I was trying to include "hadoop-minikdc" component to Yarn test framework > by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. > > <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> > The dependency inclusion seems to be working from IDE. IntelliJ picked up > the jar and did not report any compilation issue when I create a new test > program in "flink-yarn-tests" module and try to use the minikdc class. > However, when I try to run the test using maven command, I am getting > compilation error for the MiniKdc package. > Could someone please share some insights on how the dependency inclusion > works with the shaded hadoop components? > mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests > -Dtest=YARNSessionFIFOITCase > > [ERROR] COMPILATION ERROR : [INFO] > -------------------------------------------------------------[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] > package org.apache.hadoop.minikdc does not exist[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] > cannot find symbol symbol: variable MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase > RegardsVijay > > > > > > > > > |
Thanks Rob for trying. In my machine, I don't see any of minikdc related files in the target jar.
How did you run the build? RegardsVijay On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <[hidden email]> wrote: Cool, thank you. After adding the line you've mentioned, my jar contains the missing MiniKdc class: robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdcminikdc-krb5.confminikdc.ldifforg/apache/hadoop/minikdc/org/apache/hadoop/minikdc/KerberosSecurityTestcase.classorg/apache/hadoop/minikdc/MiniKdc$1.classorg/apache/hadoop/minikdc/MiniKdc.classMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xmlMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.propertiesrobert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % Is the same true on your machine? On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan <[hidden email]> wrote: Hi Rob, You need to include below lines to the pom.xml to resolve the chain dependency error. <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC dependency error --> <build> <plugins> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> <extensions>true</extensions> </plugin> </plugins> </build> RegardsVijay On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <[hidden email]> wrote: Hi, I didn't even get the shading module to build: [INFO] ------------------------------------------------------------------------[INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] Final Memory: 16M/303M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal on project flink-shaded-include-yarn-tests: Could not resolve dependencies for project org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1][ERROR] How did you do it? Regards,Robert On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan <[hidden email]> wrote: Hi Rob, Yes I checked but the result jar does not contain the classes from hadoop-minikdc package. RegardsVijay On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> wrote: Hi Vijay, did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains the required classes? Regards,Robert On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <[hidden email]> wrote: Hello, I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did not report any compilation issue when I create a new test program in "flink-yarn-tests" module and try to use the minikdc class. However, when I try to run the test using maven command, I am getting compilation error for the MiniKdc package. Could someone please share some insights on how the dependency inclusion works with the shaded hadoop components? mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase [ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] cannot find symbol symbol: variable MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] cannot find symbol symbol: class MiniKdc location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase RegardsVijay |
Hi,
This this is how I did it: robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % pwd /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % mvn clean install [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink-shaded-include-yarn-tests --- [INFO] Deleting /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target [INFO] [INFO] --- maven-checkstyle-plugin:2.16:check (validate) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ flink-shaded-include-yarn-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ flink-shaded-include-yarn-tests --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ flink-shaded-include-yarn-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ flink-shaded-include-yarn-tests --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ flink-shaded-include-yarn-tests --- [INFO] Building jar: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-shade-plugin:2.4.1:shade (shade-hadoop) @ flink-shaded-include-yarn-tests --- [INFO] Including org.apache.hadoop:hadoop-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.3.0 in the shaded jar. [INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar. [INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar. [INFO] Excluding org.apache.commons:commons-math3:jar:3.5 from the shaded jar. [INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar. [INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the shaded jar. [INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar. [INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar. [INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar. [INFO] Excluding commons-collections:commons-collections:jar:3.2.1 from the shaded jar. [INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar. [INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar. [INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-json:jar:1.9 from the shaded jar. [INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-jaxrs:jar:1.8.3 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-xc:jar:1.8.3 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar. [INFO] Including asm:asm:jar:3.1 in the shaded jar. [INFO] Excluding tomcat:jasper-compiler:jar:5.5.23 from the shaded jar. [INFO] Excluding tomcat:jasper-runtime:jar:5.5.23 from the shaded jar. [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar. [INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar. [INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar. [INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar. [INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the shaded jar. [INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the shaded jar. [INFO] Excluding com.jamesmurty.utils:java-xmlbuilder:jar:0.4 from the shaded jar. [INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar. [INFO] Excluding commons-configuration:commons-configuration:jar:1.7 from the shaded jar. [INFO] Excluding commons-digester:commons-digester:jar:1.8.1 from the shaded jar. [INFO] Excluding commons-beanutils:commons-beanutils:jar:1.8.3 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-core-asl:jar:1.8.8 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 from the shaded jar. [INFO] Excluding org.apache.avro:avro:jar:1.7.6 from the shaded jar. [INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.3 from the shaded jar. [INFO] Excluding org.xerial.snappy:snappy-java:jar:1.0.5 from the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-auth:jar:2.3.0 in the shaded jar. [INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar. [INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar. [INFO] Including io.netty:netty:jar:3.7.0.Final in the shaded jar. [INFO] Excluding org.apache.commons:commons-compress:jar:1.4.1 from the shaded jar. [INFO] Excluding org.tukaani:xz:jar:1.0 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:test-jar:tests:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-minicluster:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-common:test-jar:tests:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.3.0 in the shaded jar. [INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-jobclient:test-jar:tests:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-hs:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-minikdc:jar:2.3.0 in the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-constants:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-i18n:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-client-api:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-model:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.mina:mina-core:jar:2.0.7 from the shaded jar. [INFO] Excluding net.sf.ehcache:ehcache-core:jar:2.4.4 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptor-kerberos:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-authn:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-authz:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-changelog:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-collective:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-event:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-exception:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-journal:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-normalization:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-operational:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-referral:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-schema:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-subtree:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-trigger:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-trigger:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-protocol-shared:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-protocol-kerberos:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-shared:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-ldif-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-xdbm-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-mavibot-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.mavibot:mavibot:jar:1.0.0-M1 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-avl:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-all:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding commons-pool:commons-pool:jar:1.6 from the shaded jar. [INFO] Excluding antlr:antlr:jar:2.7.7 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-jdbm-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-protocol-ldap:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-codec-api:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-codec:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-sp:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding bouncycastle:bcprov-jdk15:jar:140 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.3.0 in the shaded jar. [INFO] Excluding com.google.inject.extensions:guice-servlet:jar:3.0 from the shaded jar. [INFO] Excluding com.google.inject:guice:jar:3.0 from the shaded jar. [INFO] Excluding javax.inject:javax.inject:jar:1 from the shaded jar. [INFO] Excluding aopalliance:aopalliance:jar:1.0 from the shaded jar. [INFO] Excluding com.sun.jersey.contribs:jersey-guice:jar:1.9 from the shaded jar. [INFO] Excluding javax.xml.bind:jaxb-api:jar:2.2.2 from the shaded jar. [INFO] Excluding javax.xml.stream:stax-api:jar:1.0-2 from the shaded jar. [INFO] Excluding javax.activation:activation:jar:1.1 from the shaded jar. [INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.3.0 in the shaded jar. [INFO] Excluding org.apache.flink:force-shading:jar:1.1-SNAPSHOT from the shaded jar. [INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar. [INFO] Excluding org.apache.commons:commons-lang3:jar:3.3.2 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar. [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar. [WARNING] hadoop-common-2.3.0-tests.jar, hadoop-common-2.3.0.jar define 37 overlapping classes: [WARNING] - org.apache.hadoop.io.serializer.avro.AvroRecord [WARNING] - org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpcProto$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$Builder [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EchoRequestProto$Builder [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProtoOrBuilder [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos [WARNING] - 27 more... [WARNING] hadoop-yarn-common-2.3.0.jar, hadoop-yarn-api-2.3.0.jar define 3 overlapping classes: [WARNING] - org.apache.hadoop.yarn.util.package-info [WARNING] - org.apache.hadoop.yarn.factories.package-info [WARNING] - org.apache.hadoop.yarn.factory.providers.package-info [WARNING] maven-shade-plugin has detected that some class files are [WARNING] present in two or more JARs. When this happens, only one [WARNING] single version of the class is copied to the uber jar. [WARNING] Usually this is not harmful and you can skip these warnings, [WARNING] otherwise try to manually exclude artifacts based on [WARNING] mvn dependency:tree -Ddetail=true and the above output. [WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar with /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT-shaded.jar [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] [INFO] --- maven-javadoc-plugin:2.9.1:jar (attach-javadocs) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (integration-tests) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- japicmp-maven-plugin:0.7.0:cmp (default) @ flink-shaded-include-yarn-tests --- [INFO] Skipping execution because parameter 'skip' was set to true. [INFO] [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ flink-shaded-include-yarn-tests --- [INFO] Installing /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar to /home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar [INFO] Installing /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml to /home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.906 s [INFO] Finished at: 2016-06-21T18:06:43+02:00 [INFO] Final Memory: 33M/420M [INFO] ------------------------------------------------------------------------ mvn clean install 55.91s user 1.63s system 354% cpu 16.229 total robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar| grep minikdc minikdc-krb5.conf minikdc.ldiff org/apache/hadoop/minikdc/ org/apache/hadoop/minikdc/KerberosSecurityTestcase.class org/apache/hadoop/minikdc/MiniKdc$1.class org/apache/hadoop/minikdc/MiniKdc.class META-INF/maven/org.apache.hadoop/hadoop-minikdc/ META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties On Tue, Jun 21, 2016 at 5:47 PM, Vijay Srinivasaraghavan < [hidden email]> wrote: > Thanks Rob for trying. In my machine, I don't see any of minikdc related > files in the target jar. > > How did you run the build? > > Regards > Vijay > > > On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <[hidden email]> > wrote: > > > Cool, thank you. > > After adding the line you've mentioned, my jar contains the missing > MiniKdc class: > > robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] > % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep > minikdc > minikdc-krb5.conf > minikdc.ldiff > org/apache/hadoop/minikdc/ > org/apache/hadoop/minikdc/KerberosSecurityTestcase.class > org/apache/hadoop/minikdc/MiniKdc$1.class > org/apache/hadoop/minikdc/MiniKdc.class > META-INF/maven/org.apache.hadoop/hadoop-minikdc/ > META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml > META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties > robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] > % > > Is the same true on your machine? > > > > > On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan < > [hidden email]> wrote: > > Hi Rob, > You need to include below lines to the pom.xml to resolve the chain > dependency error. > <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC > dependency error --> <build> <plugins> <plugin> > <groupId>org.apache.felix</groupId> > <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> > <extensions>true</extensions> </plugin> </plugins> </build> > RegardsVijay > > On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi, > I didn't even get the shading module to build: > [INFO] > ------------------------------------------------------------------------[INFO] > Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] > ------------------------------------------------------------------------[INFO] > ------------------------------------------------------------------------[INFO] > BUILD FAILURE[INFO] > ------------------------------------------------------------------------[INFO] > Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] > Final Memory: 16M/303M[INFO] > ------------------------------------------------------------------------[ERROR] > Failed to execute goal on project flink-shaded-include-yarn-tests: Could > not resolve dependencies for project > org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure > to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in > https://repo.maven.apache.org/maven2 was cached in the local repository, > resolution will not be reattempted until the update interval of central has > elapsed or updates are forced -> [Help 1][ERROR] > > How did you do it? > Regards,Robert > > On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hi Rob, > Yes I checked but the result jar does not contain the classes from > hadoop-minikdc package. > RegardsVijay > > On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi Vijay, > did you check if the artifact produced by the > "flink-shaded-include-yarn-tests" module contains the required classes? > Regards,Robert > > On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hello, > I was trying to include "hadoop-minikdc" component to Yarn test framework > by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. > > <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> > The dependency inclusion seems to be working from IDE. IntelliJ picked up > the jar and did not report any compilation issue when I create a new test > program in "flink-yarn-tests" module and try to use the minikdc class. > However, when I try to run the test using maven command, I am getting > compilation error for the MiniKdc package. > Could someone please share some insights on how the dependency inclusion > works with the shaded hadoop components? > mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests > -Dtest=YARNSessionFIFOITCase > > [ERROR] COMPILATION ERROR : [INFO] > -------------------------------------------------------------[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] > package org.apache.hadoop.minikdc does not exist[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] > cannot find symbol symbol: variable MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase > RegardsVijay > > > > > > > > > > > > > |
Hi Rob,
I ran the build and now I could see the minikdc classes bundled in "flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar" location. cd /workspace/git-projects/flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests $ jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdcminikdc-krb5.confminikdc.ldifforg/apache/hadoop/minikdc/org/apache/hadoop/minikdc/KerberosSecurityTestcase.classorg/apache/hadoop/minikdc/MiniKdc$1.classorg/apache/hadoop/minikdc/MiniKdc.classMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xmlMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties However when I run the "Yarn" integration test ("mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests"), I am getting compilation errors. I have found that the runtime classpath does not include the "flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar" from "flink-shaded-include-yarn-tests/target" location. The classpath points to local repository jar (/home/vijay/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar) which does not have the "minikdc" classes. Not sure how the classpath is derived but clearly something is missing. You could try by just importing "org.apache.hadoop.minikdc.MiniKdc" in one of the Yarn integration test class file and run "mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests" command cd ~/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT $ jar tf flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep minikdc RegardsVijay On Tuesday, June 21, 2016 9:07 AM, Robert Metzger <[hidden email]> wrote: Hi, This this is how I did it: robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % pwd /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % mvn clean install [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink-shaded-include-yarn-tests --- [INFO] Deleting /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target [INFO] [INFO] --- maven-checkstyle-plugin:2.16:check (validate) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ flink-shaded-include-yarn-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ flink-shaded-include-yarn-tests --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ flink-shaded-include-yarn-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ flink-shaded-include-yarn-tests --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ flink-shaded-include-yarn-tests --- [INFO] Building jar: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-shade-plugin:2.4.1:shade (shade-hadoop) @ flink-shaded-include-yarn-tests --- [INFO] Including org.apache.hadoop:hadoop-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.3.0 in the shaded jar. [INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar. [INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar. [INFO] Excluding org.apache.commons:commons-math3:jar:3.5 from the shaded jar. [INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar. [INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the shaded jar. [INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar. [INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar. [INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar. [INFO] Excluding commons-collections:commons-collections:jar:3.2.1 from the shaded jar. [INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar. [INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar. [INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-json:jar:1.9 from the shaded jar. [INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-jaxrs:jar:1.8.3 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-xc:jar:1.8.3 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar. [INFO] Including asm:asm:jar:3.1 in the shaded jar. [INFO] Excluding tomcat:jasper-compiler:jar:5.5.23 from the shaded jar. [INFO] Excluding tomcat:jasper-runtime:jar:5.5.23 from the shaded jar. [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar. [INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar. [INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar. [INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar. [INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the shaded jar. [INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the shaded jar. [INFO] Excluding com.jamesmurty.utils:java-xmlbuilder:jar:0.4 from the shaded jar. [INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar. [INFO] Excluding commons-configuration:commons-configuration:jar:1.7 from the shaded jar. [INFO] Excluding commons-digester:commons-digester:jar:1.8.1 from the shaded jar. [INFO] Excluding commons-beanutils:commons-beanutils:jar:1.8.3 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-core-asl:jar:1.8.8 from the shaded jar. [INFO] Excluding org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 from the shaded jar. [INFO] Excluding org.apache.avro:avro:jar:1.7.6 from the shaded jar. [INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.3 from the shaded jar. [INFO] Excluding org.xerial.snappy:snappy-java:jar:1.0.5 from the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-auth:jar:2.3.0 in the shaded jar. [INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar. [INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar. [INFO] Including io.netty:netty:jar:3.7.0.Final in the shaded jar. [INFO] Excluding org.apache.commons:commons-compress:jar:1.4.1 from the shaded jar. [INFO] Excluding org.tukaani:xz:jar:1.0 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-tests:test-jar:tests:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-minicluster:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-common:test-jar:tests:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.3.0 in the shaded jar. [INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-jobclient:test-jar:tests:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-mapreduce-client-hs:jar:2.3.0 in the shaded jar. [INFO] Including org.apache.hadoop:hadoop-minikdc:jar:2.3.0 in the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-constants:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-i18n:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-client-api:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-model:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.mina:mina-core:jar:2.0.7 from the shaded jar. [INFO] Excluding net.sf.ehcache:ehcache-core:jar:2.4.4 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptor-kerberos:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-authn:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-authz:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-changelog:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-collective:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-event:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-exception:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-journal:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-normalization:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-operational:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-referral:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-schema:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-subtree:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-interceptors-trigger:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-trigger:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-protocol-shared:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-protocol-kerberos:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-shared:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-ldif-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-xdbm-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-mavibot-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.mavibot:mavibot:jar:1.0.0-M1 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-core-avl:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-all:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding commons-pool:commons-pool:jar:1.6 from the shaded jar. [INFO] Excluding antlr:antlr:jar:2.7.7 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-jdbm-partition:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 from the shaded jar. [INFO] Excluding org.apache.directory.server:apacheds-protocol-ldap:jar:2.0.0-M15 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-codec-api:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-codec:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding org.apache.directory.api:api-ldap-extras-sp:jar:1.0.0-M20 from the shaded jar. [INFO] Excluding bouncycastle:bcprov-jdk15:jar:140 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.3.0 in the shaded jar. [INFO] Excluding com.google.inject.extensions:guice-servlet:jar:3.0 from the shaded jar. [INFO] Excluding com.google.inject:guice:jar:3.0 from the shaded jar. [INFO] Excluding javax.inject:javax.inject:jar:1 from the shaded jar. [INFO] Excluding aopalliance:aopalliance:jar:1.0 from the shaded jar. [INFO] Excluding com.sun.jersey.contribs:jersey-guice:jar:1.9 from the shaded jar. [INFO] Excluding javax.xml.bind:jaxb-api:jar:2.2.2 from the shaded jar. [INFO] Excluding javax.xml.stream:stax-api:jar:1.0-2 from the shaded jar. [INFO] Excluding javax.activation:activation:jar:1.1 from the shaded jar. [INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 from the shaded jar. [INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar. [INFO] Including org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.3.0 in the shaded jar. [INFO] Excluding org.apache.flink:force-shading:jar:1.1-SNAPSHOT from the shaded jar. [INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar. [INFO] Excluding org.apache.commons:commons-lang3:jar:3.3.2 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar. [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar. [WARNING] hadoop-common-2.3.0-tests.jar, hadoop-common-2.3.0.jar define 37 overlapping classes: [WARNING] - org.apache.hadoop.io.serializer.avro.AvroRecord [WARNING] - org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpcProto$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$Builder [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProto$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos$TestProtobufRpc2Proto$1 [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EchoRequestProto$Builder [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos$EmptyResponseProtoOrBuilder [WARNING] - org.apache.hadoop.ipc.protobuf.TestProtos [WARNING] - 27 more... [WARNING] hadoop-yarn-common-2.3.0.jar, hadoop-yarn-api-2.3.0.jar define 3 overlapping classes: [WARNING] - org.apache.hadoop.yarn.util.package-info [WARNING] - org.apache.hadoop.yarn.factories.package-info [WARNING] - org.apache.hadoop.yarn.factory.providers.package-info [WARNING] maven-shade-plugin has detected that some class files are [WARNING] present in two or more JARs. When this happens, only one [WARNING] single version of the class is copied to the uber jar. [WARNING] Usually this is not harmful and you can skip these warnings, [WARNING] otherwise try to manually exclude artifacts based on [WARNING] mvn dependency:tree -Ddetail=true and the above output. [WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar with /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT-shaded.jar [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml [INFO] [INFO] --- maven-javadoc-plugin:2.9.1:jar (attach-javadocs) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (integration-tests) @ flink-shaded-include-yarn-tests --- [INFO] [INFO] --- japicmp-maven-plugin:0.7.0:cmp (default) @ flink-shaded-include-yarn-tests --- [INFO] Skipping execution because parameter 'skip' was set to true. [INFO] [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ flink-shaded-include-yarn-tests --- [INFO] Installing /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar to /home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar [INFO] Installing /home/robert/incubator-flink/flink-shaded-hadoop/flink-shaded-include-yarn-tests/target/dependency-reduced-pom.xml to /home/robert/.m2/repository/org/apache/flink/flink-shaded-include-yarn-tests/1.1-SNAPSHOT/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.906 s [INFO] Finished at: 2016-06-21T18:06:43+02:00 [INFO] Final Memory: 33M/420M [INFO] ------------------------------------------------------------------------ mvn clean install 55.91s user 1.63s system 354% cpu 16.229 total robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar| grep minikdc minikdc-krb5.conf minikdc.ldiff org/apache/hadoop/minikdc/ org/apache/hadoop/minikdc/KerberosSecurityTestcase.class org/apache/hadoop/minikdc/MiniKdc$1.class org/apache/hadoop/minikdc/MiniKdc.class META-INF/maven/org.apache.hadoop/hadoop-minikdc/ META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties On Tue, Jun 21, 2016 at 5:47 PM, Vijay Srinivasaraghavan < [hidden email]> wrote: > Thanks Rob for trying. In my machine, I don't see any of minikdc related > files in the target jar. > > How did you run the build? > > Regards > Vijay > > > On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <[hidden email]> > wrote: > > > Cool, thank you. > > After adding the line you've mentioned, my jar contains the missing > MiniKdc class: > > robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] > % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar | grep > minikdc > minikdc-krb5.conf > minikdc.ldiff > org/apache/hadoop/minikdc/ > org/apache/hadoop/minikdc/KerberosSecurityTestcase.class > org/apache/hadoop/minikdc/MiniKdc$1.class > org/apache/hadoop/minikdc/MiniKdc.class > META-INF/maven/org.apache.hadoop/hadoop-minikdc/ > META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xml > META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.properties > robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] > % > > Is the same true on your machine? > > > > > On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan < > [hidden email]> wrote: > > Hi Rob, > You need to include below lines to the pom.xml to resolve the chain > dependency error. > <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Mini-KDC > dependency error --> <build> <plugins> <plugin> > <groupId>org.apache.felix</groupId> > <artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited> > <extensions>true</extensions> </plugin> </plugins> </build> > RegardsVijay > > On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi, > I didn't even get the shading module to build: > [INFO] > ------------------------------------------------------------------------[INFO] > Building flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] > ------------------------------------------------------------------------[INFO] > ------------------------------------------------------------------------[INFO] > BUILD FAILURE[INFO] > ------------------------------------------------------------------------[INFO] > Total time: 0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] > Final Memory: 16M/303M[INFO] > ------------------------------------------------------------------------[ERROR] > Failed to execute goal on project flink-shaded-include-yarn-tests: Could > not resolve dependencies for project > org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure > to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in > https://repo.maven.apache.org/maven2 was cached in the local repository, > resolution will not be reattempted until the update interval of central has > elapsed or updates are forced -> [Help 1][ERROR] > > How did you do it? > Regards,Robert > > On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hi Rob, > Yes I checked but the result jar does not contain the classes from > hadoop-minikdc package. > RegardsVijay > > On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <[hidden email]> > wrote: > > > Hi Vijay, > did you check if the artifact produced by the > "flink-shaded-include-yarn-tests" module contains the required classes? > Regards,Robert > > On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan > <[hidden email]> wrote: > > Hello, > I was trying to include "hadoop-minikdc" component to Yarn test framework > by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. > > <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${hadoop.version}</version></dependency> > The dependency inclusion seems to be working from IDE. IntelliJ picked up > the jar and did not report any compilation issue when I create a new test > program in "flink-yarn-tests" module and try to use the minikdc class. > However, when I try to run the test using maven command, I am getting > compilation error for the MiniKdc package. > Could someone please share some insights on how the dependency inclusion > works with the shaded hadoop components? > mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests > -Dtest=YARNSessionFIFOITCase > > [ERROR] COMPILATION ERROR : [INFO] > -------------------------------------------------------------[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33] > package org.apache.hadoop.minikdc does not exist[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38] > cannot find symbol symbol: variable MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR] > /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27] > cannot find symbol symbol: class MiniKdc location: class > org.apache.flink.yarn.YARNSessionSecuredFIFOITCase > RegardsVijay > > > > > > > > > > > > > |
This post was updated on .
Any modifications to "flink-shaded-include-yarn-tests" with respect to the dependency inclusion/exclusion does not get reflected in the "flink-yarn-tests" test case since the referenced packages from "flink-shaded-include-yarn-tests" are not being used (though the pom file establish the dependency). I am able to confirm by deleting the dependency chain from the "flink-shaded-include-yarn-tests" and rerun the "flink-yarn-tests" IT test cases. The reason why this works because of the reference to the "flink-shaded-includeyarn-tests" packages from "local" maven repository being in the classpath.
I would like to understand if this has been designed like this or is it a bug? I am not able to succesfully include a new package (hadoop-minikdc) and write new test cases to run "Yarn" test module. Any help is highly appreciated. I use below commands to run the "test" module and in case if that is incorrect please let me know the right way to run the tests. mvn clean package (to build the source and run unit test) mvn clean verify -pl flink-yarn-tests -Pinclude-yarn-tests (to run Yarn test) I also use the filter ( -Dtest=YARNSessionFIFOITCase#testJavaAPI ) to run specific test case. P.S: I have also noticed the surefire plugin is not stable and sometimes it throws some wierd issues which is very difficult to debug. Regards Vijay |
Free forum by Nabble | Edit this page |