Hey Guys,
The .getExecutionEnvironment() method for the StreamingexecutionEnvironment does not work properly because we always return LocalEnvironment for running on the minicluster. I was trying to figure out how to fix this but I got lost in the code trying to find how you determine whether the program was executed standalone or the command line client. Could you help me out with some pointers here? Regards, Gyula |
Hi,
ExecutionEnvironment has a static field contextEnvironment. When you call getExecutionEnvironment it checks whether this is null. If it is null it will return a LocalEnvironment, otherwise it will return the context environment. In Client.run() the Client creates a ContextEnvironment and calls setAsContext() on it. This will put the env into the contextEnvironment field of ExecutionEnvironment. I think you don't have to deal with all this though. Can you just wrap a normal ExecutionEnvironment inside your streaming-specific environment? In that case, you could just use ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of that. This is how it works in the Scala API. Have a look at ExecutionEnvironment.scala, it's really just a wrapper. Hope that helps. :D Cheers, Aljoscha On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <[hidden email]> wrote: > Hey Guys, > > The .getExecutionEnvironment() method for the StreamingexecutionEnvironment > does not work properly because we always return LocalEnvironment for > running on the minicluster. > > I was trying to figure out how to fix this but I got lost in the code > trying to find how you determine whether the program was executed > standalone or the command line client. > > Could you help me out with some pointers here? > > Regards, > Gyula |
Hey,
Okay, I think this is something that I might be able to use :) The problem is that the StreamingExecutionEnvironment has nothing to do with the ExecutionEnvironment (only the name is similar) because we are completely skipping the optimizer stuff at the execution. And it seemed that the context is figured out somewhere around there. So we cannot do too much with calling the method from the ExecutionEnvironment :S But I can try figuring out something with that. Cheers, Gyula On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <[hidden email]> wrote: > Hi, > ExecutionEnvironment has a static field contextEnvironment. When you > call getExecutionEnvironment it checks whether this is null. If it is > null it will return a LocalEnvironment, otherwise it will return the > context environment. > > In Client.run() the Client creates a ContextEnvironment and calls > setAsContext() on it. This will put the env into the > contextEnvironment field of ExecutionEnvironment. > > I think you don't have to deal with all this though. Can you just wrap > a normal ExecutionEnvironment inside your streaming-specific > environment? In that case, you could just use > ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of > that. This is how it works in the Scala API. Have a look at > ExecutionEnvironment.scala, it's really just a wrapper. > > Hope that helps. :D > > Cheers, > Aljoscha > > On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <[hidden email]> wrote: > > Hey Guys, > > > > The .getExecutionEnvironment() method for the > StreamingexecutionEnvironment > > does not work properly because we always return LocalEnvironment for > > running on the minicluster. > > > > I was trying to figure out how to fix this but I got lost in the code > > trying to find how you determine whether the program was executed > > standalone or the command line client. > > > > Could you help me out with some pointers here? > > > > Regards, > > Gyula > |
Ah ok, maybe you can expose methods for direct execution in the
regular ExecutionEnvironment and then use them from you wrapper. On Fri, Nov 7, 2014 at 4:18 PM, Gyula Fóra <[hidden email]> wrote: > Hey, > > Okay, I think this is something that I might be able to use :) > > The problem is that the StreamingExecutionEnvironment has nothing to do > with the ExecutionEnvironment (only the name is similar) because we are > completely skipping the optimizer stuff at the execution. And it seemed > that the context is figured out somewhere around there. > > So we cannot do too much with calling the method from the > ExecutionEnvironment :S But I can try figuring out something with that. > > Cheers, > Gyula > > On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <[hidden email]> > wrote: > >> Hi, >> ExecutionEnvironment has a static field contextEnvironment. When you >> call getExecutionEnvironment it checks whether this is null. If it is >> null it will return a LocalEnvironment, otherwise it will return the >> context environment. >> >> In Client.run() the Client creates a ContextEnvironment and calls >> setAsContext() on it. This will put the env into the >> contextEnvironment field of ExecutionEnvironment. >> >> I think you don't have to deal with all this though. Can you just wrap >> a normal ExecutionEnvironment inside your streaming-specific >> environment? In that case, you could just use >> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of >> that. This is how it works in the Scala API. Have a look at >> ExecutionEnvironment.scala, it's really just a wrapper. >> >> Hope that helps. :D >> >> Cheers, >> Aljoscha >> >> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <[hidden email]> wrote: >> > Hey Guys, >> > >> > The .getExecutionEnvironment() method for the >> StreamingexecutionEnvironment >> > does not work properly because we always return LocalEnvironment for >> > running on the minicluster. >> > >> > I was trying to figure out how to fix this but I got lost in the code >> > trying to find how you determine whether the program was executed >> > standalone or the command line client. >> > >> > Could you help me out with some pointers here? >> > >> > Regards, >> > Gyula >> |
Okay, I got it working :)
Turns out I only needed the client from the contextenvironment to get it working. Thanks for the help :) On Fri, Nov 7, 2014 at 4:26 PM, Aljoscha Krettek <[hidden email]> wrote: > Ah ok, maybe you can expose methods for direct execution in the > regular ExecutionEnvironment and then use them from you wrapper. > > On Fri, Nov 7, 2014 at 4:18 PM, Gyula Fóra <[hidden email]> wrote: > > Hey, > > > > Okay, I think this is something that I might be able to use :) > > > > The problem is that the StreamingExecutionEnvironment has nothing to do > > with the ExecutionEnvironment (only the name is similar) because we are > > completely skipping the optimizer stuff at the execution. And it seemed > > that the context is figured out somewhere around there. > > > > So we cannot do too much with calling the method from the > > ExecutionEnvironment :S But I can try figuring out something with that. > > > > Cheers, > > Gyula > > > > On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <[hidden email]> > > wrote: > > > >> Hi, > >> ExecutionEnvironment has a static field contextEnvironment. When you > >> call getExecutionEnvironment it checks whether this is null. If it is > >> null it will return a LocalEnvironment, otherwise it will return the > >> context environment. > >> > >> In Client.run() the Client creates a ContextEnvironment and calls > >> setAsContext() on it. This will put the env into the > >> contextEnvironment field of ExecutionEnvironment. > >> > >> I think you don't have to deal with all this though. Can you just wrap > >> a normal ExecutionEnvironment inside your streaming-specific > >> environment? In that case, you could just use > >> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of > >> that. This is how it works in the Scala API. Have a look at > >> ExecutionEnvironment.scala, it's really just a wrapper. > >> > >> Hope that helps. :D > >> > >> Cheers, > >> Aljoscha > >> > >> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <[hidden email]> wrote: > >> > Hey Guys, > >> > > >> > The .getExecutionEnvironment() method for the > >> StreamingexecutionEnvironment > >> > does not work properly because we always return LocalEnvironment for > >> > running on the minicluster. > >> > > >> > I was trying to figure out how to fix this but I got lost in the code > >> > trying to find how you determine whether the program was executed > >> > standalone or the command line client. > >> > > >> > Could you help me out with some pointers here? > >> > > >> > Regards, > >> > Gyula > >> > |
Nice!
On Fri, Nov 7, 2014 at 11:09 PM, Gyula Fóra <[hidden email]> wrote: > Okay, I got it working :) > Turns out I only needed the client from the contextenvironment to get it > working. > > Thanks for the help :) > > On Fri, Nov 7, 2014 at 4:26 PM, Aljoscha Krettek <[hidden email]> > wrote: > > > Ah ok, maybe you can expose methods for direct execution in the > > regular ExecutionEnvironment and then use them from you wrapper. > > > > On Fri, Nov 7, 2014 at 4:18 PM, Gyula Fóra <[hidden email]> wrote: > > > Hey, > > > > > > Okay, I think this is something that I might be able to use :) > > > > > > The problem is that the StreamingExecutionEnvironment has nothing to do > > > with the ExecutionEnvironment (only the name is similar) because we are > > > completely skipping the optimizer stuff at the execution. And it seemed > > > that the context is figured out somewhere around there. > > > > > > So we cannot do too much with calling the method from the > > > ExecutionEnvironment :S But I can try figuring out something with that. > > > > > > Cheers, > > > Gyula > > > > > > On Fri, Nov 7, 2014 at 4:00 PM, Aljoscha Krettek <[hidden email]> > > > wrote: > > > > > >> Hi, > > >> ExecutionEnvironment has a static field contextEnvironment. When you > > >> call getExecutionEnvironment it checks whether this is null. If it is > > >> null it will return a LocalEnvironment, otherwise it will return the > > >> context environment. > > >> > > >> In Client.run() the Client creates a ContextEnvironment and calls > > >> setAsContext() on it. This will put the env into the > > >> contextEnvironment field of ExecutionEnvironment. > > >> > > >> I think you don't have to deal with all this though. Can you just wrap > > >> a normal ExecutionEnvironment inside your streaming-specific > > >> environment? In that case, you could just use > > >> ExecutionEnvironemnt.getExecutionEnvironment and wrap the result of > > >> that. This is how it works in the Scala API. Have a look at > > >> ExecutionEnvironment.scala, it's really just a wrapper. > > >> > > >> Hope that helps. :D > > >> > > >> Cheers, > > >> Aljoscha > > >> > > >> On Fri, Nov 7, 2014 at 3:24 PM, Gyula Fóra <[hidden email]> wrote: > > >> > Hey Guys, > > >> > > > >> > The .getExecutionEnvironment() method for the > > >> StreamingexecutionEnvironment > > >> > does not work properly because we always return LocalEnvironment for > > >> > running on the minicluster. > > >> > > > >> > I was trying to figure out how to fix this but I got lost in the > code > > >> > trying to find how you determine whether the program was executed > > >> > standalone or the command line client. > > >> > > > >> > Could you help me out with some pointers here? > > >> > > > >> > Regards, > > >> > Gyula > > >> > > > |
Free forum by Nabble | Edit this page |