Hi guys, I am having troubles with a portion of code that throws an exception when executed on a cluster (through the web interface) but not when executed localy. The code : object DorotheaLassoRegression { def main(args: Array[String]): Unit = { val env = ExecutionEnvironment.getExecutionEnvironment val y = env.readTextFile( "hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels" ).setParallelism(1).map(x => x.toDouble) val Y = y.reduceGroup(iterator => iterator.toArray) env.execute() } } The exception: An error occurred while invoking the program: The main method caused an error. java.lang.ClassNotFoundException: eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317) at com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.accept(Unknown Source) at com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.accept(Unknown Source) at org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93) at org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113) at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42) at eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229) at org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) at java.lang.Thread.run(Thread.java:745) Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a version problem ? Thank you for your help, kind regards. Thomas ------------------------- Thomas Peel Data Scientist (M) +32 478 98 93 81 EURA NOVA Rue Emile Francqui, 4 1435 Mont-Saint-Guibert (T) +32 10 75 02 00 euranova.eu |
Hi. What flink version is running now in cluster?
I copied your code and packaging with flink quickstart archetype. There is another error because you don’t add any data sink in the program. Regards, Chiwan Park > On Jun 16, 2015, at 4:17 PM, Thomas Peel <[hidden email]> wrote: > > > > Hi guys, > > I am having troubles with a portion of code that throws an exception > when executed on a cluster (through the web interface) but not when > executed localy. > > The code : > > object DorotheaLassoRegression { > > def main(args: Array[String]): Unit = { > > val env = ExecutionEnvironment.getExecutionEnvironment > > val y = env.readTextFile( > "hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels" > ).setParallelism(1).map(x => x.toDouble) > > val Y = y.reduceGroup(iterator => iterator.toArray) > env.execute() > } > } > > The exception: > An error occurred while invoking the program: > > The main method caused an error. > > java.lang.ClassNotFoundException: > eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at > java.net.URLClassLoader$1.run(URLClassLoader.java:366) at > java.net.URLClassLoader$1.run(URLClassLoader.java:355) at > java.security.AccessController.doPrivileged(Native Method) at > java.net.URLClassLoader.findClass(URLClassLoader.java:354) at > java.lang.ClassLoader.loadClass(ClassLoader.java:425) at > java.lang.ClassLoader.loadClass(ClassLoader.java:358) at > java.lang.Class.forName0(Native Method) at > java.lang.Class.forName(Class.java:270) at > org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317) > at > com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.accept(Unknown > Source) at > com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.accept(Unknown > Source) at > org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93) > at > org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113) > at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at > org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at > org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at > eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42) > at > eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > at > org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229) > at > org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at > javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) > at > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) > at org.eclipse.jetty.server.Server.handle(Server.java:352) at > org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) > at > org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) > at > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) > at java.lang.Thread.run(Thread.java:745) > > Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a > version problem ? > > Thank you for your help, kind regards. > Thomas > > ------------------------- > Thomas Peel > > Data Scientist > (M) +32 478 98 93 81 > > EURA NOVA > > Rue Emile Francqui, 4 > > 1435 Mont-Saint-Guibert > > (T) +32 10 75 02 00 > > euranova.eu > > |
I also tested your code with an additional sink and it is executed on a
cluster without problems. Cheers, Till On Tue, Jun 16, 2015 at 10:00 AM Chiwan Park <[hidden email]> wrote: > Hi. What flink version is running now in cluster? > I copied your code and packaging with flink quickstart archetype. There is > another error because you don’t add any data sink in the program. > > Regards, > Chiwan Park > > > On Jun 16, 2015, at 4:17 PM, Thomas Peel <[hidden email]> > wrote: > > > > > > > > Hi guys, > > > > I am having troubles with a portion of code that throws an exception > > when executed on a cluster (through the web interface) but not when > > executed localy. > > > > The code : > > > > object DorotheaLassoRegression { > > > > def main(args: Array[String]): Unit = { > > > > val env = ExecutionEnvironment.getExecutionEnvironment > > > > val y = env.readTextFile( > > "hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels" > > ).setParallelism(1).map(x => x.toDouble) > > > > val Y = y.reduceGroup(iterator => iterator.toArray) > > env.execute() > > } > > } > > > > The exception: > > An error occurred while invoking the program: > > > > The main method caused an error. > > > > java.lang.ClassNotFoundException: > > eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at > > java.net.URLClassLoader$1.run(URLClassLoader.java:366) at > > java.net.URLClassLoader$1.run(URLClassLoader.java:355) at > > java.security.AccessController.doPrivileged(Native Method) at > > java.net.URLClassLoader.findClass(URLClassLoader.java:354) at > > java.lang.ClassLoader.loadClass(ClassLoader.java:425) at > > java.lang.ClassLoader.loadClass(ClassLoader.java:358) at > > java.lang.Class.forName0(Native Method) at > > java.lang.Class.forName(Class.java:270) at > > > org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317) > > at > > com.esotericsoftware.reflectasm.shaded.org > .objectweb.asm.ClassReader.accept(Unknown > > Source) at > > com.esotericsoftware.reflectasm.shaded.org > .objectweb.asm.ClassReader.accept(Unknown > > Source) at > > > org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93) > > at > > > org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113) > > at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at > > org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at > > org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at > > > eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42) > > at > > > eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) at > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > > at > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > > at > > org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229) > > at > > > org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at > > javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) > > at > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901) > > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) > > at > > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) > > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) > > at org.eclipse.jetty.server.Server.handle(Server.java:352) at > > > org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) > > at > > > org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) > > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) > > at > > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) > > at > > > org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) > > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) > > at java.lang.Thread.run(Thread.java:745) > > > > Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a > > version problem ? > > > > Thank you for your help, kind regards. > > Thomas > > > > ------------------------- > > Thomas Peel > > > > Data Scientist > > (M) +32 478 98 93 81 > > > > EURA NOVA > > > > Rue Emile Francqui, 4 > > > > 1435 Mont-Saint-Guibert > > > > (T) +32 10 75 02 00 > > > > euranova.eu > > > > > > > > |
Hi,
Thank you for your help. It was actually an issue between the Flink version I used to build my quickstart jar and the Flink version on the cluster. Cheers, Thomas Le 16/06/15 10:10, Till Rohrmann a écrit : > I also tested your code with an additional sink and it is executed on a > cluster without problems. > > Cheers, > Till > > On Tue, Jun 16, 2015 at 10:00 AM Chiwan Park <[hidden email]> wrote: > >> Hi. What flink version is running now in cluster? >> I copied your code and packaging with flink quickstart archetype. There is >> another error because you don’t add any data sink in the program. >> >> Regards, >> Chiwan Park >> >>> On Jun 16, 2015, at 4:17 PM, Thomas Peel <[hidden email]> >> wrote: >>> >>> >>> Hi guys, >>> >>> I am having troubles with a portion of code that throws an exception >>> when executed on a cluster (through the web interface) but not when >>> executed localy. >>> >>> The code : >>> >>> object DorotheaLassoRegression { >>> >>> def main(args: Array[String]): Unit = { >>> >>> val env = ExecutionEnvironment.getExecutionEnvironment >>> >>> val y = env.readTextFile( >>> "hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels" >>> ).setParallelism(1).map(x => x.toDouble) >>> >>> val Y = y.reduceGroup(iterator => iterator.toArray) >>> env.execute() >>> } >>> } >>> >>> The exception: >>> An error occurred while invoking the program: >>> >>> The main method caused an error. >>> >>> java.lang.ClassNotFoundException: >>> eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at >>> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at >>> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at >>> java.security.AccessController.doPrivileged(Native Method) at >>> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at >>> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at >>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at >>> java.lang.Class.forName0(Native Method) at >>> java.lang.Class.forName(Class.java:270) at >>> >> org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317) >>> at >>> com.esotericsoftware.reflectasm.shaded.org >> .objectweb.asm.ClassReader.accept(Unknown >>> Source) at >>> com.esotericsoftware.reflectasm.shaded.org >> .objectweb.asm.ClassReader.accept(Unknown >>> Source) at >>> >> org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93) >>> at >>> >> org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113) >>> at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at >>> org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at >>> org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at >>> >> eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42) >>> at >>> >> eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) at >>> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) >>> at >>> >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) >>> at >>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229) >>> at >>> >> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at >>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) >>> at >>> >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) >>> at >>> >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) >>> at >>> >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965) >>> at >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) >>> at >>> >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187) >>> at >>> >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901) >>> at >>> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) >>> at >>> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) >>> at >>> >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) >>> at org.eclipse.jetty.server.Server.handle(Server.java:352) at >>> >> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) >>> at >>> >> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) >>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at >>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) >>> at >>> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) >>> at >>> >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) >>> at >>> >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a >>> version problem ? >>> >>> Thank you for your help, kind regards. >>> Thomas >>> >>> ------------------------- >>> Thomas Peel >>> >>> Data Scientist >>> (M) +32 478 98 93 81 >>> >>> EURA NOVA >>> >>> Rue Emile Francqui, 4 >>> >>> 1435 Mont-Saint-Guibert >>> >>> (T) +32 10 75 02 00 >>> >>> euranova.eu >>> >>> >> >> >> |
Free forum by Nabble | Edit this page |