Running Flink Job with dependencies

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Running Flink Job with dependencies

santosh_rajaguru
Hi,

I am trying to run a Flink job, which uses the eclipse extension mechanism, from command prompt as well as from the webclient. The job runs successfully from eclipse as a java application when i configure the classpath to include the dependencies.
However, how to do that while running from webclient/ command prompt.
when i try to execute the job, it says entry point could not be loaded due to a linkage failure.
causes : NoClassDefFoundException: Org.eclipse.core.runtime

am i missing something?

Regards,
Santosh
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

Robert Metzger
Maybe you have to provide the name of the main class with the "-c" command:
./bin/flink run -c my.package.Class <args" ?

On Wed, Jun 24, 2015 at 11:28 AM, santosh_rajaguru <[hidden email]>
wrote:

> Hi,
>
> I am trying to run a Flink job, which uses the eclipse extension mechanism,
> from command prompt as well as from the webclient. The job runs
> successfully
> from eclipse as a java application when i configure the classpath to
> include
> the dependencies.
> However, how to do that while running from webclient/ command prompt.
> when i try to execute the job, it says entry point could not be loaded due
> to a linkage failure.
> causes : NoClassDefFoundException: Org.eclipse.core.runtime
>
> am i missing something?
>
> Regards,
> Santosh
>
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Running-Flink-Job-with-dependencies-tp6653.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

Stephan Ewen
This looks like the Eclipse classes are not part of the JAR. Are you
packing a fat JAR, or do you put the Eclipse dependencies in the classpath?

On Wed, Jun 24, 2015 at 11:53 AM, Robert Metzger <[hidden email]>
wrote:

> Maybe you have to provide the name of the main class with the "-c" command:
> ./bin/flink run -c my.package.Class <args" ?
>
> On Wed, Jun 24, 2015 at 11:28 AM, santosh_rajaguru <[hidden email]>
> wrote:
>
> > Hi,
> >
> > I am trying to run a Flink job, which uses the eclipse extension
> mechanism,
> > from command prompt as well as from the webclient. The job runs
> > successfully
> > from eclipse as a java application when i configure the classpath to
> > include
> > the dependencies.
> > However, how to do that while running from webclient/ command prompt.
> > when i try to execute the job, it says entry point could not be loaded
> due
> > to a linkage failure.
> > causes : NoClassDefFoundException: Org.eclipse.core.runtime
> >
> > am i missing something?
> >
> > Regards,
> > Santosh
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Running-Flink-Job-with-dependencies-tp6653.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

santosh_rajaguru
Yes, i am packaging all the dependencies to a Fat jar.
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

Flavio Pompermaier
Are you sure that the plugin dependencies are of type jar and not of type
bundle?
I remember that eclipse projects that are OSGI plugins are not managed as a
normal jar by maven usually..

On Wed, Jun 24, 2015 at 1:45 PM, santosh_rajaguru <[hidden email]> wrote:

> Yes, i am packaging all the dependencies to a Fat jar.
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Running-Flink-Job-with-dependencies-tp6653p6669.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

santosh_rajaguru
Thanks Flavio.

Previously i packaged all the Jars into Fat Jar. Despite of specifying the Main class, it was unable to load some of the classes.
Now, its running. As i have created Plugins jar and deployed those to Flink library and just created a simple jar for the main class plugin. Now, its looking from the flink library and it works fine.
In this way, i am able to use the eclipse extension point mechanism with Flink.

Cheers,
Santosh Kumar Rajaguru
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

Flavio Pompermaier
So you've put the plugin jars in the lib dir of each task manager?
It could work but I'll try to package the plugins as a jar and transitively
collect the required dependency.
However I remember that it's quite tricky to find the correct way to handle
them as a normal jars :)
Maybe some good Flink dev working with OSGI Eclipse plugins could write a
documentation about it (if any)..

On Thu, Jun 25, 2015 at 12:19 PM, santosh_rajaguru <[hidden email]>
wrote:

> Thanks Flavio.
>
> Previously i packaged all the Jars into Fat Jar. Despite of specifying the
> Main class, it was unable to load some of the classes.
> Now, its running. As i have created Plugins jar and deployed those to Flink
> library and just created a simple jar for the main class plugin. Now, its
> looking from the flink library and it works fine.
> In this way, i am able to use the eclipse extension point mechanism with
> Flink.
>
> Cheers,
> Santosh Kumar Rajaguru
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Running-Flink-Job-with-dependencies-tp6653p6699.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Running Flink Job with dependencies

santosh_rajaguru
Yes.
I have configured a product file and specified the required plugins.
It generates the jar files and i deploy them into lib folder.
and deployed the main class jar into webclient.
Ran the job.