I was trying out the new scala-shell with streaming support...
The following code executes correctly the first time I run it: val survival = benv.readCsvFile[(String, String, String, String)]("file:///home/trevor/gits/datasets/haberman/haberman.data") survival.count() However, if I call survival.count() again, run the entire code again I get the following error: java.lang.NullPointerException at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1074) at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1004) at org.apache.flink.api.java.ScalaShellRemoteEnvironment.execute(ScalaShellRemoteEnvironment.java:70) at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:898) at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:638) at org.apache.flink.api.scala.DataSet.count(DataSet.scala:528) at .<init>(<console>:62) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734) at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983) at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568) at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760) at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:805) at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:717) at scala.tools.nsc.interpreter.ILoop.processLine$1(ILoop.scala:581) at scala.tools.nsc.interpreter.ILoop.innerLoop$1(ILoop.scala:588) at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:591) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:882) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:837) at org.apache.flink.api.scala.FlinkShell$.startShell(FlinkShell.scala:199) at org.apache.flink.api.scala.FlinkShell$.main(FlinkShell.scala:127) at org.apache.flink.api.scala.FlinkShell.main(FlinkShell.scala) Is this me or is this a bug? or both? Thanks, tg Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil* |
Hi Trevor,
this is a bug and was introduced before the streaming shell: https://issues.apache.org/jira/browse/FLINK-3701 I suspect something changed somewhere else in the Flink code that breaks the scala shell. In the scala shell tests executing the same program twice is not done so it went unnoticed. The streaming shell does seem to work. If you need the batch shell, I checked for version 1.0 and it seemed to work. best, Nikolaas 2016-04-18 23:35 GMT+02:00 Trevor Grant <[hidden email]>: > I was trying out the new scala-shell with streaming support... > > The following code executes correctly the first time I run it: > > val survival = benv.readCsvFile[(String, String, String, > String)]("file:///home/trevor/gits/datasets/haberman/haberman.data") > survival.count() > > However, if I call survival.count() again, run the entire code again I get > the following error: > > java.lang.NullPointerException > at > > org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1074) > at > > org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1004) > at > > org.apache.flink.api.java.ScalaShellRemoteEnvironment.execute(ScalaShellRemoteEnvironment.java:70) > at > > org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:898) > at > > org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:638) > at org.apache.flink.api.scala.DataSet.count(DataSet.scala:528) > at .<init>(<console>:62) > at .<clinit>(<console>) > at .<init>(<console>:7) > at .<clinit>(<console>) > at $print(<console>) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734) > at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983) > at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573) > at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604) > at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568) > at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760) > at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:805) > at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:717) > at scala.tools.nsc.interpreter.ILoop.processLine$1(ILoop.scala:581) > at scala.tools.nsc.interpreter.ILoop.innerLoop$1(ILoop.scala:588) > at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:591) > at > > scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:882) > at > scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837) > at > scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837) > at > > scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) > at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:837) > at org.apache.flink.api.scala.FlinkShell$.startShell(FlinkShell.scala:199) > at org.apache.flink.api.scala.FlinkShell$.main(FlinkShell.scala:127) > at org.apache.flink.api.scala.FlinkShell.main(FlinkShell.scala) > > > Is this me or is this a bug? or both? > > Thanks, > > tg > > Trevor Grant > Data Scientist > https://github.com/rawkintrevo > http://stackexchange.com/users/3002022/rawkintrevo > http://trevorgrant.org > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > |
Hi Trevor,
As Nikolaas pointed out, this is a regression in the recent master. The current implementation of code in the ExecutionConfig doesn't expect multiple executions. We have a pending fix [1] which should be in soon. Cheers, Max [1] https://github.com/apache/flink/pull/1913 On Tue, Apr 19, 2016 at 11:58 AM, Nikolaas s <[hidden email]> wrote: > Hi Trevor, > > this is a bug and was introduced before the streaming shell: > https://issues.apache.org/jira/browse/FLINK-3701 > > I suspect something changed somewhere else in the Flink code that breaks > the scala shell. > In the scala shell tests executing the same program twice is not done so it > went unnoticed. > > The streaming shell does seem to work. If you need the batch shell, I > checked for version 1.0 and it seemed to work. > > best, > Nikolaas > > > > 2016-04-18 23:35 GMT+02:00 Trevor Grant <[hidden email]>: > >> I was trying out the new scala-shell with streaming support... >> >> The following code executes correctly the first time I run it: >> >> val survival = benv.readCsvFile[(String, String, String, >> String)]("file:///home/trevor/gits/datasets/haberman/haberman.data") >> survival.count() >> >> However, if I call survival.count() again, run the entire code again I get >> the following error: >> >> java.lang.NullPointerException >> at >> >> org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1074) >> at >> >> org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1004) >> at >> >> org.apache.flink.api.java.ScalaShellRemoteEnvironment.execute(ScalaShellRemoteEnvironment.java:70) >> at >> >> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:898) >> at >> >> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:638) >> at org.apache.flink.api.scala.DataSet.count(DataSet.scala:528) >> at .<init>(<console>:62) >> at .<clinit>(<console>) >> at .<init>(<console>:7) >> at .<clinit>(<console>) >> at $print(<console>) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734) >> at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983) >> at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573) >> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604) >> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568) >> at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760) >> at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:805) >> at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:717) >> at scala.tools.nsc.interpreter.ILoop.processLine$1(ILoop.scala:581) >> at scala.tools.nsc.interpreter.ILoop.innerLoop$1(ILoop.scala:588) >> at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:591) >> at >> >> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:882) >> at >> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837) >> at >> scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837) >> at >> >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:837) >> at org.apache.flink.api.scala.FlinkShell$.startShell(FlinkShell.scala:199) >> at org.apache.flink.api.scala.FlinkShell$.main(FlinkShell.scala:127) >> at org.apache.flink.api.scala.FlinkShell.main(FlinkShell.scala) >> >> >> Is this me or is this a bug? or both? >> >> Thanks, >> >> tg >> >> Trevor Grant >> Data Scientist >> https://github.com/rawkintrevo >> http://stackexchange.com/users/3002022/rawkintrevo >> http://trevorgrant.org >> >> *"Fortunate is he, who is able to know the causes of things." -Virgil* >> |
Free forum by Nabble | Edit this page |