Flink and Clojure

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink and Clojure

Matthias J. Sax-2
Hi Squirrels,

I was playing with a Flink Clojure WordCount example today.
https://github.com/mjsax/flink-external/tree/master/flink-clojure

After building the project with "mvn package" I tried to submit it to a
local cluster. Before I started the cluster, I manually copied
"clojure-1.5.1.jar" into Flink's lib folder.

> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar lib/
> bin/start-local.sh

However, when submitting the jar, I get an exception:

> bin/flink run -c org.apache.flink.clojure.WordCount
~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar


> ------------------------------------------------------------
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The program's entry point class 'org.apache.flink.clojure.WordCount' threw an error during initialization.
> at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> Caused by: java.lang.ExceptionInInitializerError
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:278)
> at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> ... 5 more
> Caused by: java.io.FileNotFoundException: Could not locate org/apache/flink/clojure/WordCount__init.class or org/apache/flink/clojure/WordCount.clj on classpath:
> at clojure.lang.RT.load(RT.java:443)
> at clojure.lang.RT.load(RT.java:411)
> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> at clojure.core$load.doInvoke(core.clj:5529)
> at clojure.lang.RestFn.invoke(RestFn.java:408)
> at clojure.lang.Var.invoke(Var.java:415)
> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> ... 8 more
I am not sure why the class is not found. It is contained in the jar
file. I can fix this error by copying the user jar
(flink-clojure-0.10.0.jar) into Flink's lib-folder.

It seems, that Flink is not looking into the user-jar when loading this
class. Can anybody explain why?

Thx.

-Matthias


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Aljoscha Krettek-2
Clojure is not considering the user-jar when trying to load the class.

> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]> wrote:
>
> Hi Squirrels,
>
> I was playing with a Flink Clojure WordCount example today.
> https://github.com/mjsax/flink-external/tree/master/flink-clojure
>
> After building the project with "mvn package" I tried to submit it to a
> local cluster. Before I started the cluster, I manually copied
> "clojure-1.5.1.jar" into Flink's lib folder.
>
>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar lib/
>> bin/start-local.sh
>
> However, when submitting the jar, I get an exception:
>
>> bin/flink run -c org.apache.flink.clojure.WordCount
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
>
>
>> ------------------------------------------------------------
>> The program finished with the following exception:
>>
>> org.apache.flink.client.program.ProgramInvocationException: The program's entry point class 'org.apache.flink.clojure.WordCount' threw an error during initialization.
>> at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
>> at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
>> at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
>> Caused by: java.lang.ExceptionInInitializerError
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:278)
>> at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
>> ... 5 more
>> Caused by: java.io.FileNotFoundException: Could not locate org/apache/flink/clojure/WordCount__init.class or org/apache/flink/clojure/WordCount.clj on classpath:
>> at clojure.lang.RT.load(RT.java:443)
>> at clojure.lang.RT.load(RT.java:411)
>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
>> at clojure.core$load.doInvoke(core.clj:5529)
>> at clojure.lang.RestFn.invoke(RestFn.java:408)
>> at clojure.lang.Var.invoke(Var.java:415)
>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
>> ... 8 more
>
> I am not sure why the class is not found. It is contained in the jar
> file. I can fix this error by copying the user jar
> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
>
> It seems, that Flink is not looking into the user-jar when loading this
> class. Can anybody explain why?
>
> Thx.
>
> -Matthias
>

Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Matthias J. Sax-2
Would it make sense (if possible?) for Flink to add the user jar
dynamically to it's own classpath so Clojure can find it? Or somehow
modify Clojure's class loader?

The jars in lib are added to the classpath at startup. This makes it
practically impossible to execute a Flink program that is written in
Clojure right now...


On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:

> Clojure is not considering the user-jar when trying to load the class.
>
>> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]> wrote:
>>
>> Hi Squirrels,
>>
>> I was playing with a Flink Clojure WordCount example today.
>> https://github.com/mjsax/flink-external/tree/master/flink-clojure
>>
>> After building the project with "mvn package" I tried to submit it to a
>> local cluster. Before I started the cluster, I manually copied
>> "clojure-1.5.1.jar" into Flink's lib folder.
>>
>>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar lib/
>>> bin/start-local.sh
>>
>> However, when submitting the jar, I get an exception:
>>
>>> bin/flink run -c org.apache.flink.clojure.WordCount
>> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
>>
>>
>>> ------------------------------------------------------------
>>> The program finished with the following exception:
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: The program's entry point class 'org.apache.flink.clojure.WordCount' threw an error during initialization.
>>> at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
>>> at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
>>> at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
>>> Caused by: java.lang.ExceptionInInitializerError
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:278)
>>> at org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
>>> ... 5 more
>>> Caused by: java.io.FileNotFoundException: Could not locate org/apache/flink/clojure/WordCount__init.class or org/apache/flink/clojure/WordCount.clj on classpath:
>>> at clojure.lang.RT.load(RT.java:443)
>>> at clojure.lang.RT.load(RT.java:411)
>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
>>> at clojure.core$load.doInvoke(core.clj:5529)
>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
>>> at clojure.lang.Var.invoke(Var.java:415)
>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
>>> ... 8 more
>>
>> I am not sure why the class is not found. It is contained in the jar
>> file. I can fix this error by copying the user jar
>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
>>
>> It seems, that Flink is not looking into the user-jar when loading this
>> class. Can anybody explain why?
>>
>> Thx.
>>
>> -Matthias
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Stephan Ewen
This is a problem in Java.
I think you cannot dynamically modify the initial system class loader.

What most apps do is check for the thread context class loader when
dynamically loading classes. We can check and make sure that one is set,
but if Closure does not respect that, we have a problem.
Then Closure is not built for dynamic class loading.



On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]> wrote:

> Would it make sense (if possible?) for Flink to add the user jar
> dynamically to it's own classpath so Clojure can find it? Or somehow
> modify Clojure's class loader?
>
> The jars in lib are added to the classpath at startup. This makes it
> practically impossible to execute a Flink program that is written in
> Clojure right now...
>
>
> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > Clojure is not considering the user-jar when trying to load the class.
> >
> >> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]> wrote:
> >>
> >> Hi Squirrels,
> >>
> >> I was playing with a Flink Clojure WordCount example today.
> >> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> >>
> >> After building the project with "mvn package" I tried to submit it to a
> >> local cluster. Before I started the cluster, I manually copied
> >> "clojure-1.5.1.jar" into Flink's lib folder.
> >>
> >>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar lib/
> >>> bin/start-local.sh
> >>
> >> However, when submitting the jar, I get an exception:
> >>
> >>> bin/flink run -c org.apache.flink.clojure.WordCount
> >>
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> >>
> >>
> >>> ------------------------------------------------------------
> >>> The program finished with the following exception:
> >>>
> >>> org.apache.flink.client.program.ProgramInvocationException: The
> program's entry point class 'org.apache.flink.clojure.WordCount' threw an
> error during initialization.
> >>> at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> >>> at
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> >>> at
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> >>> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> >>> Caused by: java.lang.ExceptionInInitializerError
> >>> at java.lang.Class.forName0(Native Method)
> >>> at java.lang.Class.forName(Class.java:278)
> >>> at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> >>> ... 5 more
> >>> Caused by: java.io.FileNotFoundException: Could not locate
> org/apache/flink/clojure/WordCount__init.class or
> org/apache/flink/clojure/WordCount.clj on classpath:
> >>> at clojure.lang.RT.load(RT.java:443)
> >>> at clojure.lang.RT.load(RT.java:411)
> >>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> >>> at clojure.core$load.doInvoke(core.clj:5529)
> >>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> >>> at clojure.lang.Var.invoke(Var.java:415)
> >>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> >>> ... 8 more
> >>
> >> I am not sure why the class is not found. It is contained in the jar
> >> file. I can fix this error by copying the user jar
> >> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> >>
> >> It seems, that Flink is not looking into the user-jar when loading this
> >> class. Can anybody explain why?
> >>
> >> Thx.
> >>
> >> -Matthias
> >>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Nick Dimiduk-2
What happens when you follow the packaging examples provided in the flink
quick start archetypes? These have the maven-foo required to package an
uberjar suitable for flink submission. Can you try adding that step to your
pom.xml?

On Thursday, December 10, 2015, Stephan Ewen <[hidden email]> wrote:

> This is a problem in Java.
> I think you cannot dynamically modify the initial system class loader.
>
> What most apps do is check for the thread context class loader when
> dynamically loading classes. We can check and make sure that one is set,
> but if Closure does not respect that, we have a problem.
> Then Closure is not built for dynamic class loading.
>
>
>
> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]
> <javascript:;>> wrote:
>
> > Would it make sense (if possible?) for Flink to add the user jar
> > dynamically to it's own classpath so Clojure can find it? Or somehow
> > modify Clojure's class loader?
> >
> > The jars in lib are added to the classpath at startup. This makes it
> > practically impossible to execute a Flink program that is written in
> > Clojure right now...
> >
> >
> > On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > > Clojure is not considering the user-jar when trying to load the class.
> > >
> > >> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
> <javascript:;>> wrote:
> > >>
> > >> Hi Squirrels,
> > >>
> > >> I was playing with a Flink Clojure WordCount example today.
> > >> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> > >>
> > >> After building the project with "mvn package" I tried to submit it to
> a
> > >> local cluster. Before I started the cluster, I manually copied
> > >> "clojure-1.5.1.jar" into Flink's lib folder.
> > >>
> > >>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar lib/
> > >>> bin/start-local.sh
> > >>
> > >> However, when submitting the jar, I get an exception:
> > >>
> > >>> bin/flink run -c org.apache.flink.clojure.WordCount
> > >>
> >
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> > >>
> > >>
> > >>> ------------------------------------------------------------
> > >>> The program finished with the following exception:
> > >>>
> > >>> org.apache.flink.client.program.ProgramInvocationException: The
> > program's entry point class 'org.apache.flink.clojure.WordCount' threw an
> > error during initialization.
> > >>> at
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> > >>> at
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> > >>> at
> > org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> > >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> > >>> at
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> > >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> > >>> Caused by: java.lang.ExceptionInInitializerError
> > >>> at java.lang.Class.forName0(Native Method)
> > >>> at java.lang.Class.forName(Class.java:278)
> > >>> at
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> > >>> ... 5 more
> > >>> Caused by: java.io.FileNotFoundException: Could not locate
> > org/apache/flink/clojure/WordCount__init.class or
> > org/apache/flink/clojure/WordCount.clj on classpath:
> > >>> at clojure.lang.RT.load(RT.java:443)
> > >>> at clojure.lang.RT.load(RT.java:411)
> > >>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> > >>> at clojure.core$load.doInvoke(core.clj:5529)
> > >>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> > >>> at clojure.lang.Var.invoke(Var.java:415)
> > >>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> > >>> ... 8 more
> > >>
> > >> I am not sure why the class is not found. It is contained in the jar
> > >> file. I can fix this error by copying the user jar
> > >> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> > >>
> > >> It seems, that Flink is not looking into the user-jar when loading
> this
> > >> class. Can anybody explain why?
> > >>
> > >> Thx.
> > >>
> > >> -Matthias
> > >>
> > >
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Robert Metzger
I had the same though as Nick. Maybe Leiningen allows to somehow build a
fat-jar containing the clojure standard library.

On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]> wrote:

> What happens when you follow the packaging examples provided in the flink
> quick start archetypes? These have the maven-foo required to package an
> uberjar suitable for flink submission. Can you try adding that step to your
> pom.xml?
>
> On Thursday, December 10, 2015, Stephan Ewen <[hidden email]> wrote:
>
> > This is a problem in Java.
> > I think you cannot dynamically modify the initial system class loader.
> >
> > What most apps do is check for the thread context class loader when
> > dynamically loading classes. We can check and make sure that one is set,
> > but if Closure does not respect that, we have a problem.
> > Then Closure is not built for dynamic class loading.
> >
> >
> >
> > On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]
> > <javascript:;>> wrote:
> >
> > > Would it make sense (if possible?) for Flink to add the user jar
> > > dynamically to it's own classpath so Clojure can find it? Or somehow
> > > modify Clojure's class loader?
> > >
> > > The jars in lib are added to the classpath at startup. This makes it
> > > practically impossible to execute a Flink program that is written in
> > > Clojure right now...
> > >
> > >
> > > On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > > > Clojure is not considering the user-jar when trying to load the
> class.
> > > >
> > > >> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
> > <javascript:;>> wrote:
> > > >>
> > > >> Hi Squirrels,
> > > >>
> > > >> I was playing with a Flink Clojure WordCount example today.
> > > >> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> > > >>
> > > >> After building the project with "mvn package" I tried to submit it
> to
> > a
> > > >> local cluster. Before I started the cluster, I manually copied
> > > >> "clojure-1.5.1.jar" into Flink's lib folder.
> > > >>
> > > >>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> lib/
> > > >>> bin/start-local.sh
> > > >>
> > > >> However, when submitting the jar, I get an exception:
> > > >>
> > > >>> bin/flink run -c org.apache.flink.clojure.WordCount
> > > >>
> > >
> >
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> > > >>
> > > >>
> > > >>> ------------------------------------------------------------
> > > >>> The program finished with the following exception:
> > > >>>
> > > >>> org.apache.flink.client.program.ProgramInvocationException: The
> > > program's entry point class 'org.apache.flink.clojure.WordCount' threw
> an
> > > error during initialization.
> > > >>> at
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> > > >>> at
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> > > >>> at
> > > org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> > > >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> > > >>> at
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> > > >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> > > >>> Caused by: java.lang.ExceptionInInitializerError
> > > >>> at java.lang.Class.forName0(Native Method)
> > > >>> at java.lang.Class.forName(Class.java:278)
> > > >>> at
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> > > >>> ... 5 more
> > > >>> Caused by: java.io.FileNotFoundException: Could not locate
> > > org/apache/flink/clojure/WordCount__init.class or
> > > org/apache/flink/clojure/WordCount.clj on classpath:
> > > >>> at clojure.lang.RT.load(RT.java:443)
> > > >>> at clojure.lang.RT.load(RT.java:411)
> > > >>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> > > >>> at clojure.core$load.doInvoke(core.clj:5529)
> > > >>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> > > >>> at clojure.lang.Var.invoke(Var.java:415)
> > > >>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> > > >>> ... 8 more
> > > >>
> > > >> I am not sure why the class is not found. It is contained in the jar
> > > >> file. I can fix this error by copying the user jar
> > > >> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> > > >>
> > > >> It seems, that Flink is not looking into the user-jar when loading
> > this
> > > >> class. Can anybody explain why?
> > > >>
> > > >> Thx.
> > > >>
> > > >> -Matthias
> > > >>
> > > >
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Nick Dimiduk-2
I think Mattias's project is using maven though -- there's a pom in the
project that doesn't look generated. If you want to do it from lein, maybe
my old lein-hadoop [0] plugin can help?

[0]: https://github.com/ndimiduk/lein-hadoop

On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <[hidden email]> wrote:

> I had the same though as Nick. Maybe Leiningen allows to somehow build a
> fat-jar containing the clojure standard library.
>
> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]> wrote:
>
> > What happens when you follow the packaging examples provided in the flink
> > quick start archetypes? These have the maven-foo required to package an
> > uberjar suitable for flink submission. Can you try adding that step to
> your
> > pom.xml?
> >
> > On Thursday, December 10, 2015, Stephan Ewen <[hidden email]> wrote:
> >
> > > This is a problem in Java.
> > > I think you cannot dynamically modify the initial system class loader.
> > >
> > > What most apps do is check for the thread context class loader when
> > > dynamically loading classes. We can check and make sure that one is
> set,
> > > but if Closure does not respect that, we have a problem.
> > > Then Closure is not built for dynamic class loading.
> > >
> > >
> > >
> > > On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]
> > > <javascript:;>> wrote:
> > >
> > > > Would it make sense (if possible?) for Flink to add the user jar
> > > > dynamically to it's own classpath so Clojure can find it? Or somehow
> > > > modify Clojure's class loader?
> > > >
> > > > The jars in lib are added to the classpath at startup. This makes it
> > > > practically impossible to execute a Flink program that is written in
> > > > Clojure right now...
> > > >
> > > >
> > > > On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > > > > Clojure is not considering the user-jar when trying to load the
> > class.
> > > > >
> > > > >> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
> > > <javascript:;>> wrote:
> > > > >>
> > > > >> Hi Squirrels,
> > > > >>
> > > > >> I was playing with a Flink Clojure WordCount example today.
> > > > >> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> > > > >>
> > > > >> After building the project with "mvn package" I tried to submit it
> > to
> > > a
> > > > >> local cluster. Before I started the cluster, I manually copied
> > > > >> "clojure-1.5.1.jar" into Flink's lib folder.
> > > > >>
> > > > >>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> > lib/
> > > > >>> bin/start-local.sh
> > > > >>
> > > > >> However, when submitting the jar, I get an exception:
> > > > >>
> > > > >>> bin/flink run -c org.apache.flink.clojure.WordCount
> > > > >>
> > > >
> > >
> >
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> > > > >>
> > > > >>
> > > > >>> ------------------------------------------------------------
> > > > >>> The program finished with the following exception:
> > > > >>>
> > > > >>> org.apache.flink.client.program.ProgramInvocationException: The
> > > > program's entry point class 'org.apache.flink.clojure.WordCount'
> threw
> > an
> > > > error during initialization.
> > > > >>> at
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> > > > >>> at
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> > > > >>> at
> > > >
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> > > > >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> > > > >>> at
> > > >
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> > > > >>> at
> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> > > > >>> Caused by: java.lang.ExceptionInInitializerError
> > > > >>> at java.lang.Class.forName0(Native Method)
> > > > >>> at java.lang.Class.forName(Class.java:278)
> > > > >>> at
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> > > > >>> ... 5 more
> > > > >>> Caused by: java.io.FileNotFoundException: Could not locate
> > > > org/apache/flink/clojure/WordCount__init.class or
> > > > org/apache/flink/clojure/WordCount.clj on classpath:
> > > > >>> at clojure.lang.RT.load(RT.java:443)
> > > > >>> at clojure.lang.RT.load(RT.java:411)
> > > > >>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> > > > >>> at clojure.core$load.doInvoke(core.clj:5529)
> > > > >>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> > > > >>> at clojure.lang.Var.invoke(Var.java:415)
> > > > >>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> > > > >>> ... 8 more
> > > > >>
> > > > >> I am not sure why the class is not found. It is contained in the
> jar
> > > > >> file. I can fix this error by copying the user jar
> > > > >> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> > > > >>
> > > > >> It seems, that Flink is not looking into the user-jar when loading
> > > this
> > > > >> class. Can anybody explain why?
> > > > >>
> > > > >> Thx.
> > > > >>
> > > > >> -Matthias
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Matthias J. Sax-2
Thanks for this idea.

I extended my pom to include clojure-1.5.1.jar in my program jar.
However, the problem is still there... I did some research on the
Internet, and it seems I need to mess around with Clojure's class
loading strategy...

-Matthias

On 12/10/2015 06:47 PM, Nick Dimiduk wrote:

> I think Mattias's project is using maven though -- there's a pom in the
> project that doesn't look generated. If you want to do it from lein, maybe
> my old lein-hadoop [0] plugin can help?
>
> [0]: https://github.com/ndimiduk/lein-hadoop
>
> On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <[hidden email]> wrote:
>
>> I had the same though as Nick. Maybe Leiningen allows to somehow build a
>> fat-jar containing the clojure standard library.
>>
>> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]> wrote:
>>
>>> What happens when you follow the packaging examples provided in the flink
>>> quick start archetypes? These have the maven-foo required to package an
>>> uberjar suitable for flink submission. Can you try adding that step to
>> your
>>> pom.xml?
>>>
>>> On Thursday, December 10, 2015, Stephan Ewen <[hidden email]> wrote:
>>>
>>>> This is a problem in Java.
>>>> I think you cannot dynamically modify the initial system class loader.
>>>>
>>>> What most apps do is check for the thread context class loader when
>>>> dynamically loading classes. We can check and make sure that one is
>> set,
>>>> but if Closure does not respect that, we have a problem.
>>>> Then Closure is not built for dynamic class loading.
>>>>
>>>>
>>>>
>>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]
>>>> <javascript:;>> wrote:
>>>>
>>>>> Would it make sense (if possible?) for Flink to add the user jar
>>>>> dynamically to it's own classpath so Clojure can find it? Or somehow
>>>>> modify Clojure's class loader?
>>>>>
>>>>> The jars in lib are added to the classpath at startup. This makes it
>>>>> practically impossible to execute a Flink program that is written in
>>>>> Clojure right now...
>>>>>
>>>>>
>>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
>>>>>> Clojure is not considering the user-jar when trying to load the
>>> class.
>>>>>>
>>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
>>>> <javascript:;>> wrote:
>>>>>>>
>>>>>>> Hi Squirrels,
>>>>>>>
>>>>>>> I was playing with a Flink Clojure WordCount example today.
>>>>>>> https://github.com/mjsax/flink-external/tree/master/flink-clojure
>>>>>>>
>>>>>>> After building the project with "mvn package" I tried to submit it
>>> to
>>>> a
>>>>>>> local cluster. Before I started the cluster, I manually copied
>>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
>>>>>>>
>>>>>>>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
>>> lib/
>>>>>>>> bin/start-local.sh
>>>>>>>
>>>>>>> However, when submitting the jar, I get an exception:
>>>>>>>
>>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
>>>>>>>
>>>>>
>>>>
>>>
>> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
>>>>>>>
>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> The program finished with the following exception:
>>>>>>>>
>>>>>>>> org.apache.flink.client.program.ProgramInvocationException: The
>>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
>> threw
>>> an
>>>>> error during initialization.
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
>>>>>>>> at
>>>>>
>> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
>>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>> at
>> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
>>>>>>>> Caused by: java.lang.ExceptionInInitializerError
>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>> at java.lang.Class.forName(Class.java:278)
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
>>>>>>>> ... 5 more
>>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
>>>>> org/apache/flink/clojure/WordCount__init.class or
>>>>> org/apache/flink/clojure/WordCount.clj on classpath:
>>>>>>>> at clojure.lang.RT.load(RT.java:443)
>>>>>>>> at clojure.lang.RT.load(RT.java:411)
>>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
>>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
>>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
>>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
>>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
>>>>>>>> ... 8 more
>>>>>>>
>>>>>>> I am not sure why the class is not found. It is contained in the
>> jar
>>>>>>> file. I can fix this error by copying the user jar
>>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
>>>>>>>
>>>>>>> It seems, that Flink is not looking into the user-jar when loading
>>>> this
>>>>>>> class. Can anybody explain why?
>>>>>>>
>>>>>>> Thx.
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Nick Dimiduk-2
As far as the jvm is concerned, clojure is just another library. You should
be able to package it up like any other dependency and submit the job.
That's always how it worked in Hadoop/MR anyway...

On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <[hidden email]> wrote:

> Thanks for this idea.
>
> I extended my pom to include clojure-1.5.1.jar in my program jar.
> However, the problem is still there... I did some research on the
> Internet, and it seems I need to mess around with Clojure's class
> loading strategy...
>
> -Matthias
>
> On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
> > I think Mattias's project is using maven though -- there's a pom in the
> > project that doesn't look generated. If you want to do it from lein,
> maybe
> > my old lein-hadoop [0] plugin can help?
> >
> > [0]: https://github.com/ndimiduk/lein-hadoop
> >
> > On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <[hidden email]>
> wrote:
> >
> >> I had the same though as Nick. Maybe Leiningen allows to somehow build a
> >> fat-jar containing the clojure standard library.
> >>
> >> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]>
> wrote:
> >>
> >>> What happens when you follow the packaging examples provided in the
> flink
> >>> quick start archetypes? These have the maven-foo required to package an
> >>> uberjar suitable for flink submission. Can you try adding that step to
> >> your
> >>> pom.xml?
> >>>
> >>> On Thursday, December 10, 2015, Stephan Ewen <[hidden email]> wrote:
> >>>
> >>>> This is a problem in Java.
> >>>> I think you cannot dynamically modify the initial system class loader.
> >>>>
> >>>> What most apps do is check for the thread context class loader when
> >>>> dynamically loading classes. We can check and make sure that one is
> >> set,
> >>>> but if Closure does not respect that, we have a problem.
> >>>> Then Closure is not built for dynamic class loading.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]
> >>>> <javascript:;>> wrote:
> >>>>
> >>>>> Would it make sense (if possible?) for Flink to add the user jar
> >>>>> dynamically to it's own classpath so Clojure can find it? Or somehow
> >>>>> modify Clojure's class loader?
> >>>>>
> >>>>> The jars in lib are added to the classpath at startup. This makes it
> >>>>> practically impossible to execute a Flink program that is written in
> >>>>> Clojure right now...
> >>>>>
> >>>>>
> >>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> >>>>>> Clojure is not considering the user-jar when trying to load the
> >>> class.
> >>>>>>
> >>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
> >>>> <javascript:;>> wrote:
> >>>>>>>
> >>>>>>> Hi Squirrels,
> >>>>>>>
> >>>>>>> I was playing with a Flink Clojure WordCount example today.
> >>>>>>> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> >>>>>>>
> >>>>>>> After building the project with "mvn package" I tried to submit it
> >>> to
> >>>> a
> >>>>>>> local cluster. Before I started the cluster, I manually copied
> >>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
> >>>>>>>
> >>>>>>>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> >>> lib/
> >>>>>>>> bin/start-local.sh
> >>>>>>>
> >>>>>>> However, when submitting the jar, I get an exception:
> >>>>>>>
> >>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> >>>>>>>
> >>>>>>>
> >>>>>>>> ------------------------------------------------------------
> >>>>>>>> The program finished with the following exception:
> >>>>>>>>
> >>>>>>>> org.apache.flink.client.program.ProgramInvocationException: The
> >>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
> >> threw
> >>> an
> >>>>> error during initialization.
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> >>>>>>>> at
> >>>>>
> >> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> >>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> >>>>>>>> at
> >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> >>>>>>>> Caused by: java.lang.ExceptionInInitializerError
> >>>>>>>> at java.lang.Class.forName0(Native Method)
> >>>>>>>> at java.lang.Class.forName(Class.java:278)
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> >>>>>>>> ... 5 more
> >>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
> >>>>> org/apache/flink/clojure/WordCount__init.class or
> >>>>> org/apache/flink/clojure/WordCount.clj on classpath:
> >>>>>>>> at clojure.lang.RT.load(RT.java:443)
> >>>>>>>> at clojure.lang.RT.load(RT.java:411)
> >>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> >>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
> >>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> >>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
> >>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> >>>>>>>> ... 8 more
> >>>>>>>
> >>>>>>> I am not sure why the class is not found. It is contained in the
> >> jar
> >>>>>>> file. I can fix this error by copying the user jar
> >>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> >>>>>>>
> >>>>>>> It seems, that Flink is not looking into the user-jar when loading
> >>>> this
> >>>>>>> class. Can anybody explain why?
> >>>>>>>
> >>>>>>> Thx.
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Stephan Ewen
Flink's classloading is different from Hadoop's.

In Hadoop, the entire JVM is started with all classes (including the user
jar) in the classpath already. In Flink, jars are added dymanically, to
running JVMs with custom class loaders. That way, running worker/master
processes can accept new jars without restarts. Important for low-latency,
shells, etc

Flink itself respects these classloaders whenever dynamically looking up a
class. It may be that Closure is written such that it can only dynamically
instantiate what is the original classpath.



On Fri, Dec 11, 2015 at 1:31 AM, Nick Dimiduk <[hidden email]> wrote:

> As far as the jvm is concerned, clojure is just another library. You should
> be able to package it up like any other dependency and submit the job.
> That's always how it worked in Hadoop/MR anyway...
>
> On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <[hidden email]> wrote:
>
> > Thanks for this idea.
> >
> > I extended my pom to include clojure-1.5.1.jar in my program jar.
> > However, the problem is still there... I did some research on the
> > Internet, and it seems I need to mess around with Clojure's class
> > loading strategy...
> >
> > -Matthias
> >
> > On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
> > > I think Mattias's project is using maven though -- there's a pom in the
> > > project that doesn't look generated. If you want to do it from lein,
> > maybe
> > > my old lein-hadoop [0] plugin can help?
> > >
> > > [0]: https://github.com/ndimiduk/lein-hadoop
> > >
> > > On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <[hidden email]>
> > wrote:
> > >
> > >> I had the same though as Nick. Maybe Leiningen allows to somehow
> build a
> > >> fat-jar containing the clojure standard library.
> > >>
> > >> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]>
> > wrote:
> > >>
> > >>> What happens when you follow the packaging examples provided in the
> > flink
> > >>> quick start archetypes? These have the maven-foo required to package
> an
> > >>> uberjar suitable for flink submission. Can you try adding that step
> to
> > >> your
> > >>> pom.xml?
> > >>>
> > >>> On Thursday, December 10, 2015, Stephan Ewen <[hidden email]>
> wrote:
> > >>>
> > >>>> This is a problem in Java.
> > >>>> I think you cannot dynamically modify the initial system class
> loader.
> > >>>>
> > >>>> What most apps do is check for the thread context class loader when
> > >>>> dynamically loading classes. We can check and make sure that one is
> > >> set,
> > >>>> but if Closure does not respect that, we have a problem.
> > >>>> Then Closure is not built for dynamic class loading.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <[hidden email]
> > >>>> <javascript:;>> wrote:
> > >>>>
> > >>>>> Would it make sense (if possible?) for Flink to add the user jar
> > >>>>> dynamically to it's own classpath so Clojure can find it? Or
> somehow
> > >>>>> modify Clojure's class loader?
> > >>>>>
> > >>>>> The jars in lib are added to the classpath at startup. This makes
> it
> > >>>>> practically impossible to execute a Flink program that is written
> in
> > >>>>> Clojure right now...
> > >>>>>
> > >>>>>
> > >>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > >>>>>> Clojure is not considering the user-jar when trying to load the
> > >>> class.
> > >>>>>>
> > >>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
> > >>>> <javascript:;>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Squirrels,
> > >>>>>>>
> > >>>>>>> I was playing with a Flink Clojure WordCount example today.
> > >>>>>>>
> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> > >>>>>>>
> > >>>>>>> After building the project with "mvn package" I tried to submit
> it
> > >>> to
> > >>>> a
> > >>>>>>> local cluster. Before I started the cluster, I manually copied
> > >>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
> > >>>>>>>
> > >>>>>>>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> > >>> lib/
> > >>>>>>>> bin/start-local.sh
> > >>>>>>>
> > >>>>>>> However, when submitting the jar, I get an exception:
> > >>>>>>>
> > >>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> ------------------------------------------------------------
> > >>>>>>>> The program finished with the following exception:
> > >>>>>>>>
> > >>>>>>>> org.apache.flink.client.program.ProgramInvocationException: The
> > >>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
> > >> threw
> > >>> an
> > >>>>> error during initialization.
> > >>>>>>>> at
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> > >>>>>>>> at
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> > >>>>>>>> at
> > >>>>>
> > >> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> > >>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> > >>>>>>>> at
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> > >>>>>>>> at
> > >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> > >>>>>>>> Caused by: java.lang.ExceptionInInitializerError
> > >>>>>>>> at java.lang.Class.forName0(Native Method)
> > >>>>>>>> at java.lang.Class.forName(Class.java:278)
> > >>>>>>>> at
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> > >>>>>>>> ... 5 more
> > >>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
> > >>>>> org/apache/flink/clojure/WordCount__init.class or
> > >>>>> org/apache/flink/clojure/WordCount.clj on classpath:
> > >>>>>>>> at clojure.lang.RT.load(RT.java:443)
> > >>>>>>>> at clojure.lang.RT.load(RT.java:411)
> > >>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> > >>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
> > >>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> > >>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
> > >>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> > >>>>>>>> ... 8 more
> > >>>>>>>
> > >>>>>>> I am not sure why the class is not found. It is contained in the
> > >> jar
> > >>>>>>> file. I can fix this error by copying the user jar
> > >>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> > >>>>>>>
> > >>>>>>> It seems, that Flink is not looking into the user-jar when
> loading
> > >>>> this
> > >>>>>>> class. Can anybody explain why?
> > >>>>>>>
> > >>>>>>> Thx.
> > >>>>>>>
> > >>>>>>> -Matthias
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Nick Dimiduk-2
Ah I see. This explains the issues I had with submitting streaming jobs
that package JDBC drivers. Is there a second in the guide/docs about
classloader considerations with Flink?

On Thu, Dec 10, 2015 at 11:53 PM, Stephan Ewen <[hidden email]> wrote:

> Flink's classloading is different from Hadoop's.
>
> In Hadoop, the entire JVM is started with all classes (including the user
> jar) in the classpath already. In Flink, jars are added dymanically, to
> running JVMs with custom class loaders. That way, running worker/master
> processes can accept new jars without restarts. Important for low-latency,
> shells, etc
>
> Flink itself respects these classloaders whenever dynamically looking up a
> class. It may be that Closure is written such that it can only dynamically
> instantiate what is the original classpath.
>
>
>
> On Fri, Dec 11, 2015 at 1:31 AM, Nick Dimiduk <[hidden email]> wrote:
>
> > As far as the jvm is concerned, clojure is just another library. You
> should
> > be able to package it up like any other dependency and submit the job.
> > That's always how it worked in Hadoop/MR anyway...
> >
> > On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <[hidden email]>
> wrote:
> >
> > > Thanks for this idea.
> > >
> > > I extended my pom to include clojure-1.5.1.jar in my program jar.
> > > However, the problem is still there... I did some research on the
> > > Internet, and it seems I need to mess around with Clojure's class
> > > loading strategy...
> > >
> > > -Matthias
> > >
> > > On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
> > > > I think Mattias's project is using maven though -- there's a pom in
> the
> > > > project that doesn't look generated. If you want to do it from lein,
> > > maybe
> > > > my old lein-hadoop [0] plugin can help?
> > > >
> > > > [0]: https://github.com/ndimiduk/lein-hadoop
> > > >
> > > > On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <[hidden email]
> >
> > > wrote:
> > > >
> > > >> I had the same though as Nick. Maybe Leiningen allows to somehow
> > build a
> > > >> fat-jar containing the clojure standard library.
> > > >>
> > > >> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]>
> > > wrote:
> > > >>
> > > >>> What happens when you follow the packaging examples provided in the
> > > flink
> > > >>> quick start archetypes? These have the maven-foo required to
> package
> > an
> > > >>> uberjar suitable for flink submission. Can you try adding that step
> > to
> > > >> your
> > > >>> pom.xml?
> > > >>>
> > > >>> On Thursday, December 10, 2015, Stephan Ewen <[hidden email]>
> > wrote:
> > > >>>
> > > >>>> This is a problem in Java.
> > > >>>> I think you cannot dynamically modify the initial system class
> > loader.
> > > >>>>
> > > >>>> What most apps do is check for the thread context class loader
> when
> > > >>>> dynamically loading classes. We can check and make sure that one
> is
> > > >> set,
> > > >>>> but if Closure does not respect that, we have a problem.
> > > >>>> Then Closure is not built for dynamic class loading.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <
> [hidden email]
> > > >>>> <javascript:;>> wrote:
> > > >>>>
> > > >>>>> Would it make sense (if possible?) for Flink to add the user jar
> > > >>>>> dynamically to it's own classpath so Clojure can find it? Or
> > somehow
> > > >>>>> modify Clojure's class loader?
> > > >>>>>
> > > >>>>> The jars in lib are added to the classpath at startup. This makes
> > it
> > > >>>>> practically impossible to execute a Flink program that is written
> > in
> > > >>>>> Clojure right now...
> > > >>>>>
> > > >>>>>
> > > >>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > > >>>>>> Clojure is not considering the user-jar when trying to load the
> > > >>> class.
> > > >>>>>>
> > > >>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
> > > >>>> <javascript:;>> wrote:
> > > >>>>>>>
> > > >>>>>>> Hi Squirrels,
> > > >>>>>>>
> > > >>>>>>> I was playing with a Flink Clojure WordCount example today.
> > > >>>>>>>
> > https://github.com/mjsax/flink-external/tree/master/flink-clojure
> > > >>>>>>>
> > > >>>>>>> After building the project with "mvn package" I tried to submit
> > it
> > > >>> to
> > > >>>> a
> > > >>>>>>> local cluster. Before I started the cluster, I manually copied
> > > >>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
> > > >>>>>>>
> > > >>>>>>>> cp
> ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> > > >>> lib/
> > > >>>>>>>> bin/start-local.sh
> > > >>>>>>>
> > > >>>>>>> However, when submitting the jar, I get an exception:
> > > >>>>>>>
> > > >>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> ------------------------------------------------------------
> > > >>>>>>>> The program finished with the following exception:
> > > >>>>>>>>
> > > >>>>>>>> org.apache.flink.client.program.ProgramInvocationException:
> The
> > > >>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
> > > >> threw
> > > >>> an
> > > >>>>> error during initialization.
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> > > >>>>>>>> at
> > > >>>>>
> > > >>
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> > > >>>>>>>> at
> org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> > > >>>>>>>> at
> > > >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> > > >>>>>>>> Caused by: java.lang.ExceptionInInitializerError
> > > >>>>>>>> at java.lang.Class.forName0(Native Method)
> > > >>>>>>>> at java.lang.Class.forName(Class.java:278)
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> > > >>>>>>>> ... 5 more
> > > >>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
> > > >>>>> org/apache/flink/clojure/WordCount__init.class or
> > > >>>>> org/apache/flink/clojure/WordCount.clj on classpath:
> > > >>>>>>>> at clojure.lang.RT.load(RT.java:443)
> > > >>>>>>>> at clojure.lang.RT.load(RT.java:411)
> > > >>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> > > >>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
> > > >>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> > > >>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
> > > >>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> > > >>>>>>>> ... 8 more
> > > >>>>>>>
> > > >>>>>>> I am not sure why the class is not found. It is contained in
> the
> > > >> jar
> > > >>>>>>> file. I can fix this error by copying the user jar
> > > >>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> > > >>>>>>>
> > > >>>>>>> It seems, that Flink is not looking into the user-jar when
> > loading
> > > >>>> this
> > > >>>>>>> class. Can anybody explain why?
> > > >>>>>>>
> > > >>>>>>> Thx.
> > > >>>>>>>
> > > >>>>>>> -Matthias
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink and Clojure

Matthias J. Sax-2
Hi,

I had a closer look into this and opened an PR to fix the issue:
https://github.com/apache/flink/pull/1457

As I am afraid of side effects I am not aware of, please give feedback
if this fix can be merged or not...

Thx.

-Matthias

On 12/11/2015 06:26 PM, Nick Dimiduk wrote:

> Ah I see. This explains the issues I had with submitting streaming jobs
> that package JDBC drivers. Is there a second in the guide/docs about
> classloader considerations with Flink?
>
> On Thu, Dec 10, 2015 at 11:53 PM, Stephan Ewen <[hidden email]> wrote:
>
>> Flink's classloading is different from Hadoop's.
>>
>> In Hadoop, the entire JVM is started with all classes (including the user
>> jar) in the classpath already. In Flink, jars are added dymanically, to
>> running JVMs with custom class loaders. That way, running worker/master
>> processes can accept new jars without restarts. Important for low-latency,
>> shells, etc
>>
>> Flink itself respects these classloaders whenever dynamically looking up a
>> class. It may be that Closure is written such that it can only dynamically
>> instantiate what is the original classpath.
>>
>>
>>
>> On Fri, Dec 11, 2015 at 1:31 AM, Nick Dimiduk <[hidden email]> wrote:
>>
>>> As far as the jvm is concerned, clojure is just another library. You
>> should
>>> be able to package it up like any other dependency and submit the job.
>>> That's always how it worked in Hadoop/MR anyway...
>>>
>>> On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <[hidden email]>
>> wrote:
>>>
>>>> Thanks for this idea.
>>>>
>>>> I extended my pom to include clojure-1.5.1.jar in my program jar.
>>>> However, the problem is still there... I did some research on the
>>>> Internet, and it seems I need to mess around with Clojure's class
>>>> loading strategy...
>>>>
>>>> -Matthias
>>>>
>>>> On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
>>>>> I think Mattias's project is using maven though -- there's a pom in
>> the
>>>>> project that doesn't look generated. If you want to do it from lein,
>>>> maybe
>>>>> my old lein-hadoop [0] plugin can help?
>>>>>
>>>>> [0]: https://github.com/ndimiduk/lein-hadoop
>>>>>
>>>>> On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <[hidden email]
>>>
>>>> wrote:
>>>>>
>>>>>> I had the same though as Nick. Maybe Leiningen allows to somehow
>>> build a
>>>>>> fat-jar containing the clojure standard library.
>>>>>>
>>>>>> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <[hidden email]>
>>>> wrote:
>>>>>>
>>>>>>> What happens when you follow the packaging examples provided in the
>>>> flink
>>>>>>> quick start archetypes? These have the maven-foo required to
>> package
>>> an
>>>>>>> uberjar suitable for flink submission. Can you try adding that step
>>> to
>>>>>> your
>>>>>>> pom.xml?
>>>>>>>
>>>>>>> On Thursday, December 10, 2015, Stephan Ewen <[hidden email]>
>>> wrote:
>>>>>>>
>>>>>>>> This is a problem in Java.
>>>>>>>> I think you cannot dynamically modify the initial system class
>>> loader.
>>>>>>>>
>>>>>>>> What most apps do is check for the thread context class loader
>> when
>>>>>>>> dynamically loading classes. We can check and make sure that one
>> is
>>>>>> set,
>>>>>>>> but if Closure does not respect that, we have a problem.
>>>>>>>> Then Closure is not built for dynamic class loading.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <
>> [hidden email]
>>>>>>>> <javascript:;>> wrote:
>>>>>>>>
>>>>>>>>> Would it make sense (if possible?) for Flink to add the user jar
>>>>>>>>> dynamically to it's own classpath so Clojure can find it? Or
>>> somehow
>>>>>>>>> modify Clojure's class loader?
>>>>>>>>>
>>>>>>>>> The jars in lib are added to the classpath at startup. This makes
>>> it
>>>>>>>>> practically impossible to execute a Flink program that is written
>>> in
>>>>>>>>> Clojure right now...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
>>>>>>>>>> Clojure is not considering the user-jar when trying to load the
>>>>>>> class.
>>>>>>>>>>
>>>>>>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <[hidden email]
>>>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Squirrels,
>>>>>>>>>>>
>>>>>>>>>>> I was playing with a Flink Clojure WordCount example today.
>>>>>>>>>>>
>>> https://github.com/mjsax/flink-external/tree/master/flink-clojure
>>>>>>>>>>>
>>>>>>>>>>> After building the project with "mvn package" I tried to submit
>>> it
>>>>>>> to
>>>>>>>> a
>>>>>>>>>>> local cluster. Before I started the cluster, I manually copied
>>>>>>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
>>>>>>>>>>>
>>>>>>>>>>>> cp
>> ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
>>>>>>> lib/
>>>>>>>>>>>> bin/start-local.sh
>>>>>>>>>>>
>>>>>>>>>>> However, when submitting the jar, I get an exception:
>>>>>>>>>>>
>>>>>>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>> The program finished with the following exception:
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.client.program.ProgramInvocationException:
>> The
>>>>>>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
>>>>>> threw
>>>>>>> an
>>>>>>>>> error during initialization.
>>>>>>>>>>>> at
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
>>>>>>>>>>>> at
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
>>>>>>>>>>>> at
>>>>>>>>>
>>>>>>
>> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
>>>>>>>>>>>> at
>> org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
>>>>>>>>>>>> at
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>>>>>> at
>>>>>> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
>>>>>>>>>>>> Caused by: java.lang.ExceptionInInitializerError
>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>> at java.lang.Class.forName(Class.java:278)
>>>>>>>>>>>> at
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
>>>>>>>>>>>> ... 5 more
>>>>>>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
>>>>>>>>> org/apache/flink/clojure/WordCount__init.class or
>>>>>>>>> org/apache/flink/clojure/WordCount.clj on classpath:
>>>>>>>>>>>> at clojure.lang.RT.load(RT.java:443)
>>>>>>>>>>>> at clojure.lang.RT.load(RT.java:411)
>>>>>>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
>>>>>>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
>>>>>>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
>>>>>>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
>>>>>>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
>>>>>>>>>>>> ... 8 more
>>>>>>>>>>>
>>>>>>>>>>> I am not sure why the class is not found. It is contained in
>> the
>>>>>> jar
>>>>>>>>>>> file. I can fix this error by copying the user jar
>>>>>>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
>>>>>>>>>>>
>>>>>>>>>>> It seems, that Flink is not looking into the user-jar when
>>> loading
>>>>>>>> this
>>>>>>>>>>> class. Can anybody explain why?
>>>>>>>>>>>
>>>>>>>>>>> Thx.
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>


signature.asc (836 bytes) Download Attachment