Some new problems about flink-storm-compatibility

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Some new problems about flink-storm-compatibility

huangwei (G)
Hi,

I got some new problems about the storm compatibility currently.
These occurred when I ran the “storm-wordcount” in the storm compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink project.


First, I start a local flink:

$ cd bin

$ ./start-local.sh



Then I ran the example:

$ bin/flink run --jarfile ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal



It took a trouble like following:

Executing StormWordCount* example with built-in default data

  Provide parameters to read input data from a file

  Usage: StormWordCount* <text path> <result path>

java.lang.ExceptionInInitializerError

at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106)

at org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78)

at org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)

at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)

at org.apache.flink.client.program.Client.run(Client.java:327)

at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)

at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)

at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)

at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)

Caused by: java.lang.RuntimeException: Could not find config file on classpath defaults.yaml

at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)

at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)

at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)

at backtype.storm.utils.Utils.<clinit>(Utils.java:71)

... 14 more



And then I added the default.yaml into flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.

It took another issue in the client log:

18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming WordCount) changed to FAILED.

java.lang.Exception: Call to registerInputOutput() of invokable failed

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper

ClassLoader info: URL ClassLoader:

Class not resolvable through given classloader.

at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219)

at org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)

... 1 more



I have no idea to fix this and need some help.



Thanks!



Greetings,
Huang Wei
华为技术有限公司 Huawei Technologies Co., Ltd.


Tel:+86 18106512602
Email:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Some new problems about flink-storm-compatibility

Aljoscha Krettek-2
Hi,
could you please try running StormWordCountRemoteBySubmitter instead of
StormWordCountLocal. I think the Local one only works when running inside
an IDE (or executing by hand on the command line). The RemoteBySubmitter
variant should be the correct one for executing on a cluster using
"bin/flink".

Please get back to us if it still doesn't work or if you have further
questions.

Greetings,
Aljoscha

On Thu, 20 Aug 2015 at 13:19 huangwei (G) <[hidden email]> wrote:

> Hi,
>
> I got some new problems about the storm compatibility currently.
> These occurred when I ran the “storm-wordcount” in the storm compatibility
> on a flink-0.10SNAPSHOT which I built it over a latest flink project.
>
>
> First, I start a local flink:
>
> $ cd bin
>
> $ ./start-local.sh
>
>
>
> Then I ran the example:
>
> $ bin/flink run --jarfile
> ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c
> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal
>
>
>
> It took a trouble like following:
>
> Executing StormWordCount* example with built-in default data
>
>   Provide parameters to read input data from a file
>
>   Usage: StormWordCount* <text path> <result path>
>
> java.lang.ExceptionInInitializerError
>
> at
> backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106)
>
> at
> org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78)
>
> at
> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>
> at org.apache.flink.client.program.Client.run(Client.java:327)
>
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
>
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
>
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
>
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
>
> Caused by: java.lang.RuntimeException: Could not find config file on
> classpath defaults.yaml
>
> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
>
> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)
>
> at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)
>
> at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
>
> ... 14 more
>
>
>
> And then I added the default.yaml into
> flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.
>
> It took another issue in the client log:
>
> 18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager
>       - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming
> WordCount) changed to FAILED.
>
> java.lang.Exception: Call to registerInputOutput() of invokable failed
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
> Cannot load user class:
> org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper
>
> ClassLoader info: URL ClassLoader:
>
> Class not resolvable through given classloader.
>
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)
>
> ... 1 more
>
>
>
> I have no idea to fix this and need some help.
>
>
>
> Thanks!
>
>
>
> Greetings,
> Huang Wei
> 华为技术有限公司 Huawei Technologies Co., Ltd.
>
>
> Tel:+86 18106512602
> Email:[hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some new problems about flink-storm-compatibility

Matthias J. Sax
In reply to this post by huangwei (G)
Hi Huang,

you are using the wrong jar file. The current built, does not assemble a
jar file for StormWordCountLocal. You can extend pom.xml. Look at the
example StormWordCountRemoteBySubmitter and the corresponding assembly
file word-count-storm.xml.

What you need to do is:
  - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter
   -> change descriptor file, program-class, and id
 - copy  word-count-storm.xml to new descriptor file and adjust it
accordingly (needs to include StormWordCountLocal instead of
StormWordCountRemoteBySubmitter (and maybe other adjustments)
 - run maven -DskipTests package to build the new jar file

If you have further problems, just let us know.

-Matthias


On 08/20/2015 01:19 PM, huangwei (G) wrote:

> Hi,
>
> I got some new problems about the storm compatibility currently.
> These occurred when I ran the “storm-wordcount” in the storm compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink project.
>
>
> First, I start a local flink:
>
> $ cd bin
>
> $ ./start-local.sh
>
>
>
> Then I ran the example:
>
> $ bin/flink run --jarfile ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal
>
>
>
> It took a trouble like following:
>
> Executing StormWordCount* example with built-in default data
>
>   Provide parameters to read input data from a file
>
>   Usage: StormWordCount* <text path> <result path>
>
> java.lang.ExceptionInInitializerError
>
> at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106)
>
> at org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78)
>
> at org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>
> at org.apache.flink.client.program.Client.run(Client.java:327)
>
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
>
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
>
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
>
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
>
> Caused by: java.lang.RuntimeException: Could not find config file on classpath defaults.yaml
>
> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
>
> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)
>
> at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)
>
> at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
>
> ... 14 more
>
>
>
> And then I added the default.yaml into flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.
>
> It took another issue in the client log:
>
> 18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming WordCount) changed to FAILED.
>
> java.lang.Exception: Call to registerInputOutput() of invokable failed
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper
>
> ClassLoader info: URL ClassLoader:
>
> Class not resolvable through given classloader.
>
> at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219)
>
> at org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)
>
> ... 1 more
>
>
>
> I have no idea to fix this and need some help.
>
>
>
> Thanks!
>
>
>
> Greetings,
> Huang Wei
> 华为技术有限公司 Huawei Technologies Co., Ltd.
>
>
> Tel:+86 18106512602
> Email:[hidden email]
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Some new problems about flink-storm-compatibility

Aljoscha Krettek-2
Hi,
I'm afraid submitting the correct program also doesn't work right now. When
I try to execute it I get this:
bin/flink run --jarfile
/Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar
  -c
org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter
/Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:327)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
Caused by: NotAliveException(msg:null)
at
org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196)
at
org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190)
at
org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
... 6 more

The exception above occurred while trying to run your command.

and no output is produced. Hopefully I'm doing something wrong. Any ideas
Matthias?

By the way, the StormWordCountLocal works when executing in the IDE.

Greetings,
Aljoscha

On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <[hidden email]>
wrote:

> Hi Huang,
>
> you are using the wrong jar file. The current built, does not assemble a
> jar file for StormWordCountLocal. You can extend pom.xml. Look at the
> example StormWordCountRemoteBySubmitter and the corresponding assembly
> file word-count-storm.xml.
>
> What you need to do is:
>   - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter
>    -> change descriptor file, program-class, and id
>  - copy  word-count-storm.xml to new descriptor file and adjust it
> accordingly (needs to include StormWordCountLocal instead of
> StormWordCountRemoteBySubmitter (and maybe other adjustments)
>  - run maven -DskipTests package to build the new jar file
>
> If you have further problems, just let us know.
>
> -Matthias
>
>
> On 08/20/2015 01:19 PM, huangwei (G) wrote:
> > Hi,
> >
> > I got some new problems about the storm compatibility currently.
> > These occurred when I ran the “storm-wordcount” in the storm
> compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink
> project.
> >
> >
> > First, I start a local flink:
> >
> > $ cd bin
> >
> > $ ./start-local.sh
> >
> >
> >
> > Then I ran the example:
> >
> > $ bin/flink run --jarfile
> ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c
> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal
> >
> >
> >
> > It took a trouble like following:
> >
> > Executing StormWordCount* example with built-in default data
> >
> >   Provide parameters to read input data from a file
> >
> >   Usage: StormWordCount* <text path> <result path>
> >
> > java.lang.ExceptionInInitializerError
> >
> > at
> backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106)
> >
> > at
> org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78)
> >
> > at
> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:497)
> >
> > at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> >
> > at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> >
> > at org.apache.flink.client.program.Client.run(Client.java:327)
> >
> > at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
> >
> > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
> >
> > at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
> >
> > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
> >
> > Caused by: java.lang.RuntimeException: Could not find config file on
> classpath defaults.yaml
> >
> > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
> >
> > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)
> >
> > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)
> >
> > at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
> >
> > ... 14 more
> >
> >
> >
> > And then I added the default.yaml into
> flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.
> >
> > It took another issue in the client log:
> >
> > 18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager
>         - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming
> WordCount) changed to FAILED.
> >
> > java.lang.Exception: Call to registerInputOutput() of invokable failed
> >
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
> Cannot load user class:
> org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper
> >
> > ClassLoader info: URL ClassLoader:
> >
> > Class not resolvable through given classloader.
> >
> > at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219)
> >
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135)
> >
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)
> >
> > ... 1 more
> >
> >
> >
> > I have no idea to fix this and need some help.
> >
> >
> >
> > Thanks!
> >
> >
> >
> > Greetings,
> > Huang Wei
> > 华为技术有限公司 Huawei Technologies Co., Ltd.
> >
> >
> > Tel:+86 18106512602
> > Email:[hidden email]
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some new problems about flink-storm-compatibility

Matthias J. Sax
Hi,

I just had a look into this. Currently, not all jars are build
correctly. StormWordCountRemoteBySubmitter is build correctly though.

It starts a program, sleeps for 5 seconds and tries to kill the program
afterwards. The exception you see is "NotAliveException", meaning that
there is no program running. Did you kill it manually during the 5
second sleep phase?

Furthermore, Client.killTopology(...) is currently broken (due to
session ID changes). Thus, in StormWordCountRemoteBySubmitter example,
this need to be removed. Killing via CLI or WebUI "cancel" should work.

Fixes for those issues are already included in both of my current PRs. I
will also start a new JIRA and PR to fix up the packaging problems.
Recently, many new examples got added including package restructuring
which broke the jar building process. Need to be more careful in the
future not to break them...


-Matthias


On 08/20/2015 01:59 PM, Aljoscha Krettek wrote:

> Hi,
> I'm afraid submitting the correct program also doesn't work right now. When
> I try to execute it I get this:
> bin/flink run --jarfile
> /Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar
>   -c
> org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter
> /Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> at org.apache.flink.client.program.Client.run(Client.java:327)
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
> Caused by: NotAliveException(msg:null)
> at
> org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196)
> at
> org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190)
> at
> org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> ... 6 more
>
> The exception above occurred while trying to run your command.
>
> and no output is produced. Hopefully I'm doing something wrong. Any ideas
> Matthias?
>
> By the way, the StormWordCountLocal works when executing in the IDE.
>
> Greetings,
> Aljoscha
>
> On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <[hidden email]>
> wrote:
>
>> Hi Huang,
>>
>> you are using the wrong jar file. The current built, does not assemble a
>> jar file for StormWordCountLocal. You can extend pom.xml. Look at the
>> example StormWordCountRemoteBySubmitter and the corresponding assembly
>> file word-count-storm.xml.
>>
>> What you need to do is:
>>   - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter
>>    -> change descriptor file, program-class, and id
>>  - copy  word-count-storm.xml to new descriptor file and adjust it
>> accordingly (needs to include StormWordCountLocal instead of
>> StormWordCountRemoteBySubmitter (and maybe other adjustments)
>>  - run maven -DskipTests package to build the new jar file
>>
>> If you have further problems, just let us know.
>>
>> -Matthias
>>
>>
>> On 08/20/2015 01:19 PM, huangwei (G) wrote:
>>> Hi,
>>>
>>> I got some new problems about the storm compatibility currently.
>>> These occurred when I ran the “storm-wordcount” in the storm
>> compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink
>> project.
>>>
>>>
>>> First, I start a local flink:
>>>
>>> $ cd bin
>>>
>>> $ ./start-local.sh
>>>
>>>
>>>
>>> Then I ran the example:
>>>
>>> $ bin/flink run --jarfile
>> ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c
>> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal
>>>
>>>
>>>
>>> It took a trouble like following:
>>>
>>> Executing StormWordCount* example with built-in default data
>>>
>>>   Provide parameters to read input data from a file
>>>
>>>   Usage: StormWordCount* <text path> <result path>
>>>
>>> java.lang.ExceptionInInitializerError
>>>
>>> at
>> backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106)
>>>
>>> at
>> org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78)
>>>
>>> at
>> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>
>>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>>>
>>> at
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>>>
>>> at org.apache.flink.client.program.Client.run(Client.java:327)
>>>
>>> at
>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
>>>
>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
>>>
>>> at
>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
>>>
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
>>>
>>> Caused by: java.lang.RuntimeException: Could not find config file on
>> classpath defaults.yaml
>>>
>>> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
>>>
>>> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)
>>>
>>> at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)
>>>
>>> at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
>>>
>>> ... 14 more
>>>
>>>
>>>
>>> And then I added the default.yaml into
>> flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.
>>>
>>> It took another issue in the client log:
>>>
>>> 18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>         - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming
>> WordCount) changed to FAILED.
>>>
>>> java.lang.Exception: Call to registerInputOutput() of invokable failed
>>>
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
>> Cannot load user class:
>> org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper
>>>
>>> ClassLoader info: URL ClassLoader:
>>>
>>> Class not resolvable through given classloader.
>>>
>>> at
>> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219)
>>>
>>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135)
>>>
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)
>>>
>>> ... 1 more
>>>
>>>
>>>
>>> I have no idea to fix this and need some help.
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>> Greetings,
>>> Huang Wei
>>> 华为技术有限公司 Huawei Technologies Co., Ltd.
>>>
>>>
>>> Tel:+86 18106512602
>>> Email:[hidden email]
>>>
>>
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Some new problems about flink-storm-compatibility

huangwei (G)
In reply to this post by Aljoscha Krettek-2
Hi Aljoscha,

Once, I had the problem like yours.
What I did is to set the taskmanager.numberOfTaskSlots: 4 in conf/flink-conf.yaml since the parallelism of StormWordCountRemoteBySubmitter is 4 by default.
Maybe you can take a try.

BTW: You are right. The StormWordCountLocal gets access in IDEA.

Hi Matthias,
Thank you for the help.
I`ll try.

===========================

Hi,
I'm afraid submitting the correct program also doesn't work right now. When I try to execute it I get this:
bin/flink run --jarfile
/Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar
  -c
org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter
/Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:327)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
Caused by: NotAliveException(msg:null)
at
org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196)
at
org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190)
at
org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
... 6 more

The exception above occurred while trying to run your command.

and no output is produced. Hopefully I'm doing something wrong. Any ideas Matthias?

By the way, the StormWordCountLocal works when executing in the IDE.

Greetings,
Aljoscha

On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <[hidden email]>
wrote:

> Hi Huang,
>
> you are using the wrong jar file. The current built, does not assemble
> a jar file for StormWordCountLocal. You can extend pom.xml. Look at
> the example StormWordCountRemoteBySubmitter and the corresponding
> assembly file word-count-storm.xml.
>
> What you need to do is:
>   - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter
>    -> change descriptor file, program-class, and id
>  - copy  word-count-storm.xml to new descriptor file and adjust it
> accordingly (needs to include StormWordCountLocal instead of
> StormWordCountRemoteBySubmitter (and maybe other adjustments)
>  - run maven -DskipTests package to build the new jar file
>
> If you have further problems, just let us know.
>
> -Matthias
>
>
> On 08/20/2015 01:19 PM, huangwei (G) wrote:
> > Hi,
> >
> > I got some new problems about the storm compatibility currently.
> > These occurred when I ran the “storm-wordcount” in the storm
> compatibility on a flink-0.10SNAPSHOT which I built it over a latest
> flink project.
> >
> >
> > First, I start a local flink:
> >
> > $ cd bin
> >
> > $ ./start-local.sh
> >
> >
> >
> > Then I ran the example:
> >
> > $ bin/flink run --jarfile
> ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c
> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal
> >
> >
> >
> > It took a trouble like following:
> >
> > Executing StormWordCount* example with built-in default data
> >
> >   Provide parameters to read input data from a file
> >
> >   Usage: StormWordCount* <text path> <result path>
> >
> > java.lang.ExceptionInInitializerError
> >
> > at
> backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder
> .java:106)
> >
> > at
> org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTop
> ology(FlinkTopologyBuilder.java:78)
> >
> > at
> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main
> (StormWordCountLocal.java:66)
> >
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:62)
> >
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:497)
> >
> > at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(Package
> dProgram.java:437)
> >
> > at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeF
> orExecution(PackagedProgram.java:353)
> >
> > at org.apache.flink.client.program.Client.run(Client.java:327)
> >
> > at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:60
> 8)
> >
> > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
> >
> > at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:9
> 27)
> >
> > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
> >
> > Caused by: java.lang.RuntimeException: Could not find config file on
> classpath defaults.yaml
> >
> > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
> >
> > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)
> >
> > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)
> >
> > at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
> >
> > ... 14 more
> >
> >
> >
> > And then I added the default.yaml into
> flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.
> >
> > It took another issue in the client log:
> >
> > 18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager
>         - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming
> WordCount) changed to FAILED.
> >
> > java.lang.Exception: Call to registerInputOutput() of invokable
> > failed
> >
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
> Cannot load user class:
> org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper
> >
> > ClassLoader info: URL ClassLoader:
> >
> > Class not resolvable through given classloader.
> >
> > at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(St
> reamConfig.java:219)
> >
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutpu
> t(StreamTask.java:135)
> >
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)
> >
> > ... 1 more
> >
> >
> >
> > I have no idea to fix this and need some help.
> >
> >
> >
> > Thanks!
> >
> >
> >
> > Greetings,
> > Huang Wei
> > 华为技术有限公司 Huawei Technologies Co., Ltd.
> >
> >
> > Tel:+86 18106512602
> > Email:[hidden email]
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some new problems about flink-storm-compatibility

Aljoscha Krettek-2
Thanks, changing the numerOfTaskSlots did it.

On Thu, 20 Aug 2015 at 15:08 huangwei (G) <[hidden email]> wrote:

> Hi Aljoscha,
>
> Once, I had the problem like yours.
> What I did is to set the taskmanager.numberOfTaskSlots: 4 in
> conf/flink-conf.yaml since the parallelism of
> StormWordCountRemoteBySubmitter is 4 by default.
> Maybe you can take a try.
>
> BTW: You are right. The StormWordCountLocal gets access in IDEA.
>
> Hi Matthias,
> Thank you for the help.
> I`ll try.
>
> ===========================
>
> Hi,
> I'm afraid submitting the correct program also doesn't work right now.
> When I try to execute it I get this:
> bin/flink run --jarfile
>
> /Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar
>   -c
>
> org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter
> /Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error.
> at
>
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> at
>
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> at org.apache.flink.client.program.Client.run(Client.java:327)
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
> Caused by: NotAliveException(msg:null)
> at
>
> org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196)
> at
>
> org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190)
> at
>
> org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
>
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> ... 6 more
>
> The exception above occurred while trying to run your command.
>
> and no output is produced. Hopefully I'm doing something wrong. Any ideas
> Matthias?
>
> By the way, the StormWordCountLocal works when executing in the IDE.
>
> Greetings,
> Aljoscha
>
> On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <
> [hidden email]>
> wrote:
>
> > Hi Huang,
> >
> > you are using the wrong jar file. The current built, does not assemble
> > a jar file for StormWordCountLocal. You can extend pom.xml. Look at
> > the example StormWordCountRemoteBySubmitter and the corresponding
> > assembly file word-count-storm.xml.
> >
> > What you need to do is:
> >   - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter
> >    -> change descriptor file, program-class, and id
> >  - copy  word-count-storm.xml to new descriptor file and adjust it
> > accordingly (needs to include StormWordCountLocal instead of
> > StormWordCountRemoteBySubmitter (and maybe other adjustments)
> >  - run maven -DskipTests package to build the new jar file
> >
> > If you have further problems, just let us know.
> >
> > -Matthias
> >
> >
> > On 08/20/2015 01:19 PM, huangwei (G) wrote:
> > > Hi,
> > >
> > > I got some new problems about the storm compatibility currently.
> > > These occurred when I ran the “storm-wordcount” in the storm
> > compatibility on a flink-0.10SNAPSHOT which I built it over a latest
> > flink project.
> > >
> > >
> > > First, I start a local flink:
> > >
> > > $ cd bin
> > >
> > > $ ./start-local.sh
> > >
> > >
> > >
> > > Then I ran the example:
> > >
> > > $ bin/flink run --jarfile
> > ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c
> > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal
> > >
> > >
> > >
> > > It took a trouble like following:
> > >
> > > Executing StormWordCount* example with built-in default data
> > >
> > >   Provide parameters to read input data from a file
> > >
> > >   Usage: StormWordCount* <text path> <result path>
> > >
> > > java.lang.ExceptionInInitializerError
> > >
> > > at
> > backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder
> > .java:106)
> > >
> > > at
> > org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTop
> > ology(FlinkTopologyBuilder.java:78)
> > >
> > > at
> > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main
> > (StormWordCountLocal.java:66)
> > >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >
> > > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:62)
> > >
> > > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:43)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:497)
> > >
> > > at
> > org.apache.flink.client.program.PackagedProgram.callMainMethod(Package
> > dProgram.java:437)
> > >
> > > at
> > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeF
> > orExecution(PackagedProgram.java:353)
> > >
> > > at org.apache.flink.client.program.Client.run(Client.java:327)
> > >
> > > at
> > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:60
> > 8)
> > >
> > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296)
> > >
> > > at
> > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:9
> > 27)
> > >
> > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977)
> > >
> > > Caused by: java.lang.RuntimeException: Could not find config file on
> > classpath defaults.yaml
> > >
> > > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129)
> > >
> > > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160)
> > >
> > > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184)
> > >
> > > at backtype.storm.utils.Utils.<clinit>(Utils.java:71)
> > >
> > > ... 14 more
> > >
> > >
> > >
> > > And then I added the default.yaml into
> > flink-storm-compatibility-examples-0.10-SNAPSHOT.jar.
> > >
> > > It took another issue in the client log:
> > >
> > > 18:55:42,526 INFO  org.apache.flink.runtime.jobmanager.JobManager
> >         - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming
> > WordCount) changed to FAILED.
> > >
> > > java.lang.Exception: Call to registerInputOutput() of invokable
> > > failed
> > >
> > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Caused by:
> org.apache.flink.streaming.runtime.tasks.StreamTaskException:
> > Cannot load user class:
> > org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper
> > >
> > > ClassLoader info: URL ClassLoader:
> > >
> > > Class not resolvable through given classloader.
> > >
> > > at
> > org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(St
> > reamConfig.java:219)
> > >
> > > at
> > org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutpu
> > t(StreamTask.java:135)
> > >
> > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523)
> > >
> > > ... 1 more
> > >
> > >
> > >
> > > I have no idea to fix this and need some help.
> > >
> > >
> > >
> > > Thanks!
> > >
> > >
> > >
> > > Greetings,
> > > Huang Wei
> > > 华为技术有限公司 Huawei Technologies Co., Ltd.
> > >
> > >
> > > Tel:+86 18106512602
> > > Email:[hidden email]
> > >
> >
> >
>