Hi,
I got some new problems about the storm compatibility currently. These occurred when I ran the “storm-wordcount” in the storm compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink project. First, I start a local flink: $ cd bin $ ./start-local.sh Then I ran the example: $ bin/flink run --jarfile ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal It took a trouble like following: Executing StormWordCount* example with built-in default data Provide parameters to read input data from a file Usage: StormWordCount* <text path> <result path> java.lang.ExceptionInInitializerError at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106) at org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78) at org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) at org.apache.flink.client.program.Client.run(Client.java:327) at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) Caused by: java.lang.RuntimeException: Could not find config file on classpath defaults.yaml at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) at backtype.storm.utils.Utils.<clinit>(Utils.java:71) ... 14 more And then I added the default.yaml into flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. It took another issue in the client log: 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming WordCount) changed to FAILED. java.lang.Exception: Call to registerInputOutput() of invokable failed at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper ClassLoader info: URL ClassLoader: Class not resolvable through given classloader. at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219) at org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) ... 1 more I have no idea to fix this and need some help. Thanks! Greetings, Huang Wei 华为技术有限公司 Huawei Technologies Co., Ltd. Tel:+86 18106512602 Email:[hidden email] |
Hi,
could you please try running StormWordCountRemoteBySubmitter instead of StormWordCountLocal. I think the Local one only works when running inside an IDE (or executing by hand on the command line). The RemoteBySubmitter variant should be the correct one for executing on a cluster using "bin/flink". Please get back to us if it still doesn't work or if you have further questions. Greetings, Aljoscha On Thu, 20 Aug 2015 at 13:19 huangwei (G) <[hidden email]> wrote: > Hi, > > I got some new problems about the storm compatibility currently. > These occurred when I ran the “storm-wordcount” in the storm compatibility > on a flink-0.10SNAPSHOT which I built it over a latest flink project. > > > First, I start a local flink: > > $ cd bin > > $ ./start-local.sh > > > > Then I ran the example: > > $ bin/flink run --jarfile > ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal > > > > It took a trouble like following: > > Executing StormWordCount* example with built-in default data > > Provide parameters to read input data from a file > > Usage: StormWordCount* <text path> <result path> > > java.lang.ExceptionInInitializerError > > at > backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106) > > at > org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78) > > at > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > > at org.apache.flink.client.program.Client.run(Client.java:327) > > at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > > Caused by: java.lang.RuntimeException: Could not find config file on > classpath defaults.yaml > > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) > > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) > > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) > > at backtype.storm.utils.Utils.<clinit>(Utils.java:71) > > ... 14 more > > > > And then I added the default.yaml into > flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. > > It took another issue in the client log: > > 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager > - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming > WordCount) changed to FAILED. > > java.lang.Exception: Call to registerInputOutput() of invokable failed > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: > Cannot load user class: > org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper > > ClassLoader info: URL ClassLoader: > > Class not resolvable through given classloader. > > at > org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219) > > at > org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135) > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) > > ... 1 more > > > > I have no idea to fix this and need some help. > > > > Thanks! > > > > Greetings, > Huang Wei > 华为技术有限公司 Huawei Technologies Co., Ltd. > > > Tel:+86 18106512602 > Email:[hidden email] > > |
In reply to this post by huangwei (G)
Hi Huang,
you are using the wrong jar file. The current built, does not assemble a jar file for StormWordCountLocal. You can extend pom.xml. Look at the example StormWordCountRemoteBySubmitter and the corresponding assembly file word-count-storm.xml. What you need to do is: - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter -> change descriptor file, program-class, and id - copy word-count-storm.xml to new descriptor file and adjust it accordingly (needs to include StormWordCountLocal instead of StormWordCountRemoteBySubmitter (and maybe other adjustments) - run maven -DskipTests package to build the new jar file If you have further problems, just let us know. -Matthias On 08/20/2015 01:19 PM, huangwei (G) wrote: > Hi, > > I got some new problems about the storm compatibility currently. > These occurred when I ran the “storm-wordcount” in the storm compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink project. > > > First, I start a local flink: > > $ cd bin > > $ ./start-local.sh > > > > Then I ran the example: > > $ bin/flink run --jarfile ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal > > > > It took a trouble like following: > > Executing StormWordCount* example with built-in default data > > Provide parameters to read input data from a file > > Usage: StormWordCount* <text path> <result path> > > java.lang.ExceptionInInitializerError > > at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106) > > at org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78) > > at org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > > at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > > at org.apache.flink.client.program.Client.run(Client.java:327) > > at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > > at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > > Caused by: java.lang.RuntimeException: Could not find config file on classpath defaults.yaml > > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) > > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) > > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) > > at backtype.storm.utils.Utils.<clinit>(Utils.java:71) > > ... 14 more > > > > And then I added the default.yaml into flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. > > It took another issue in the client log: > > 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming WordCount) changed to FAILED. > > java.lang.Exception: Call to registerInputOutput() of invokable failed > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper > > ClassLoader info: URL ClassLoader: > > Class not resolvable through given classloader. > > at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219) > > at org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135) > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) > > ... 1 more > > > > I have no idea to fix this and need some help. > > > > Thanks! > > > > Greetings, > Huang Wei > 华为技术有限公司 Huawei Technologies Co., Ltd. > > > Tel:+86 18106512602 > Email:[hidden email] > |
Hi,
I'm afraid submitting the correct program also doesn't work right now. When I try to execute it I get this: bin/flink run --jarfile /Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar -c org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter /Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) at org.apache.flink.client.program.Client.run(Client.java:327) at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) Caused by: NotAliveException(msg:null) at org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196) at org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190) at org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) ... 6 more The exception above occurred while trying to run your command. and no output is produced. Hopefully I'm doing something wrong. Any ideas Matthias? By the way, the StormWordCountLocal works when executing in the IDE. Greetings, Aljoscha On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <[hidden email]> wrote: > Hi Huang, > > you are using the wrong jar file. The current built, does not assemble a > jar file for StormWordCountLocal. You can extend pom.xml. Look at the > example StormWordCountRemoteBySubmitter and the corresponding assembly > file word-count-storm.xml. > > What you need to do is: > - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter > -> change descriptor file, program-class, and id > - copy word-count-storm.xml to new descriptor file and adjust it > accordingly (needs to include StormWordCountLocal instead of > StormWordCountRemoteBySubmitter (and maybe other adjustments) > - run maven -DskipTests package to build the new jar file > > If you have further problems, just let us know. > > -Matthias > > > On 08/20/2015 01:19 PM, huangwei (G) wrote: > > Hi, > > > > I got some new problems about the storm compatibility currently. > > These occurred when I ran the “storm-wordcount” in the storm > compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink > project. > > > > > > First, I start a local flink: > > > > $ cd bin > > > > $ ./start-local.sh > > > > > > > > Then I ran the example: > > > > $ bin/flink run --jarfile > ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal > > > > > > > > It took a trouble like following: > > > > Executing StormWordCount* example with built-in default data > > > > Provide parameters to read input data from a file > > > > Usage: StormWordCount* <text path> <result path> > > > > java.lang.ExceptionInInitializerError > > > > at > backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106) > > > > at > org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78) > > > > at > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > > > > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > > > > at org.apache.flink.client.program.Client.run(Client.java:327) > > > > at > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > > > > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) > > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > > > > Caused by: java.lang.RuntimeException: Could not find config file on > classpath defaults.yaml > > > > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) > > > > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) > > > > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) > > > > at backtype.storm.utils.Utils.<clinit>(Utils.java:71) > > > > ... 14 more > > > > > > > > And then I added the default.yaml into > flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. > > > > It took another issue in the client log: > > > > 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager > - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming > WordCount) changed to FAILED. > > > > java.lang.Exception: Call to registerInputOutput() of invokable failed > > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) > > > > at java.lang.Thread.run(Thread.java:745) > > > > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: > Cannot load user class: > org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper > > > > ClassLoader info: URL ClassLoader: > > > > Class not resolvable through given classloader. > > > > at > org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219) > > > > at > org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135) > > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) > > > > ... 1 more > > > > > > > > I have no idea to fix this and need some help. > > > > > > > > Thanks! > > > > > > > > Greetings, > > Huang Wei > > 华为技术有限公司 Huawei Technologies Co., Ltd. > > > > > > Tel:+86 18106512602 > > Email:[hidden email] > > > > |
Hi,
I just had a look into this. Currently, not all jars are build correctly. StormWordCountRemoteBySubmitter is build correctly though. It starts a program, sleeps for 5 seconds and tries to kill the program afterwards. The exception you see is "NotAliveException", meaning that there is no program running. Did you kill it manually during the 5 second sleep phase? Furthermore, Client.killTopology(...) is currently broken (due to session ID changes). Thus, in StormWordCountRemoteBySubmitter example, this need to be removed. Killing via CLI or WebUI "cancel" should work. Fixes for those issues are already included in both of my current PRs. I will also start a new JIRA and PR to fix up the packaging problems. Recently, many new examples got added including package restructuring which broke the jar building process. Need to be more careful in the future not to break them... -Matthias On 08/20/2015 01:59 PM, Aljoscha Krettek wrote: > Hi, > I'm afraid submitting the correct program also doesn't work right now. When > I try to execute it I get this: > bin/flink run --jarfile > /Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar > -c > org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter > /Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error. > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > at org.apache.flink.client.program.Client.run(Client.java:327) > at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > Caused by: NotAliveException(msg:null) > at > org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196) > at > org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190) > at > org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > ... 6 more > > The exception above occurred while trying to run your command. > > and no output is produced. Hopefully I'm doing something wrong. Any ideas > Matthias? > > By the way, the StormWordCountLocal works when executing in the IDE. > > Greetings, > Aljoscha > > On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <[hidden email]> > wrote: > >> Hi Huang, >> >> you are using the wrong jar file. The current built, does not assemble a >> jar file for StormWordCountLocal. You can extend pom.xml. Look at the >> example StormWordCountRemoteBySubmitter and the corresponding assembly >> file word-count-storm.xml. >> >> What you need to do is: >> - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter >> -> change descriptor file, program-class, and id >> - copy word-count-storm.xml to new descriptor file and adjust it >> accordingly (needs to include StormWordCountLocal instead of >> StormWordCountRemoteBySubmitter (and maybe other adjustments) >> - run maven -DskipTests package to build the new jar file >> >> If you have further problems, just let us know. >> >> -Matthias >> >> >> On 08/20/2015 01:19 PM, huangwei (G) wrote: >>> Hi, >>> >>> I got some new problems about the storm compatibility currently. >>> These occurred when I ran the “storm-wordcount” in the storm >> compatibility on a flink-0.10SNAPSHOT which I built it over a latest flink >> project. >>> >>> >>> First, I start a local flink: >>> >>> $ cd bin >>> >>> $ ./start-local.sh >>> >>> >>> >>> Then I ran the example: >>> >>> $ bin/flink run --jarfile >> ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c >> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal >>> >>> >>> >>> It took a trouble like following: >>> >>> Executing StormWordCount* example with built-in default data >>> >>> Provide parameters to read input data from a file >>> >>> Usage: StormWordCount* <text path> <result path> >>> >>> java.lang.ExceptionInInitializerError >>> >>> at >> backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:106) >>> >>> at >> org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTopology(FlinkTopologyBuilder.java:78) >>> >>> at >> org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main(StormWordCountLocal.java:66) >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >>> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> >>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) >>> >>> at >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) >>> >>> at org.apache.flink.client.program.Client.run(Client.java:327) >>> >>> at >> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) >>> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) >>> >>> at >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) >>> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) >>> >>> Caused by: java.lang.RuntimeException: Could not find config file on >> classpath defaults.yaml >>> >>> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) >>> >>> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) >>> >>> at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) >>> >>> at backtype.storm.utils.Utils.<clinit>(Utils.java:71) >>> >>> ... 14 more >>> >>> >>> >>> And then I added the default.yaml into >> flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. >>> >>> It took another issue in the client log: >>> >>> 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager >> - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming >> WordCount) changed to FAILED. >>> >>> java.lang.Exception: Call to registerInputOutput() of invokable failed >>> >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: >> Cannot load user class: >> org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper >>> >>> ClassLoader info: URL ClassLoader: >>> >>> Class not resolvable through given classloader. >>> >>> at >> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:219) >>> >>> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutput(StreamTask.java:135) >>> >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) >>> >>> ... 1 more >>> >>> >>> >>> I have no idea to fix this and need some help. >>> >>> >>> >>> Thanks! >>> >>> >>> >>> Greetings, >>> Huang Wei >>> 华为技术有限公司 Huawei Technologies Co., Ltd. >>> >>> >>> Tel:+86 18106512602 >>> Email:[hidden email] >>> >> >> > |
In reply to this post by Aljoscha Krettek-2
Hi Aljoscha,
Once, I had the problem like yours. What I did is to set the taskmanager.numberOfTaskSlots: 4 in conf/flink-conf.yaml since the parallelism of StormWordCountRemoteBySubmitter is 4 by default. Maybe you can take a try. BTW: You are right. The StormWordCountLocal gets access in IDEA. Hi Matthias, Thank you for the help. I`ll try. =========================== Hi, I'm afraid submitting the correct program also doesn't work right now. When I try to execute it I get this: bin/flink run --jarfile /Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar -c org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter /Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) at org.apache.flink.client.program.Client.run(Client.java:327) at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) Caused by: NotAliveException(msg:null) at org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196) at org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190) at org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) ... 6 more The exception above occurred while trying to run your command. and no output is produced. Hopefully I'm doing something wrong. Any ideas Matthias? By the way, the StormWordCountLocal works when executing in the IDE. Greetings, Aljoscha On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax <[hidden email]> wrote: > Hi Huang, > > you are using the wrong jar file. The current built, does not assemble > a jar file for StormWordCountLocal. You can extend pom.xml. Look at > the example StormWordCountRemoteBySubmitter and the corresponding > assembly file word-count-storm.xml. > > What you need to do is: > - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter > -> change descriptor file, program-class, and id > - copy word-count-storm.xml to new descriptor file and adjust it > accordingly (needs to include StormWordCountLocal instead of > StormWordCountRemoteBySubmitter (and maybe other adjustments) > - run maven -DskipTests package to build the new jar file > > If you have further problems, just let us know. > > -Matthias > > > On 08/20/2015 01:19 PM, huangwei (G) wrote: > > Hi, > > > > I got some new problems about the storm compatibility currently. > > These occurred when I ran the “storm-wordcount” in the storm > compatibility on a flink-0.10SNAPSHOT which I built it over a latest > flink project. > > > > > > First, I start a local flink: > > > > $ cd bin > > > > $ ./start-local.sh > > > > > > > > Then I ran the example: > > > > $ bin/flink run --jarfile > ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal > > > > > > > > It took a trouble like following: > > > > Executing StormWordCount* example with built-in default data > > > > Provide parameters to read input data from a file > > > > Usage: StormWordCount* <text path> <result path> > > > > java.lang.ExceptionInInitializerError > > > > at > backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder > .java:106) > > > > at > org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTop > ology(FlinkTopologyBuilder.java:78) > > > > at > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main > (StormWordCountLocal.java:66) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > ava:62) > > > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > orImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(Package > dProgram.java:437) > > > > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeF > orExecution(PackagedProgram.java:353) > > > > at org.apache.flink.client.program.Client.run(Client.java:327) > > > > at > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:60 > 8) > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > > > > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:9 > 27) > > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > > > > Caused by: java.lang.RuntimeException: Could not find config file on > classpath defaults.yaml > > > > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) > > > > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) > > > > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) > > > > at backtype.storm.utils.Utils.<clinit>(Utils.java:71) > > > > ... 14 more > > > > > > > > And then I added the default.yaml into > flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. > > > > It took another issue in the client log: > > > > 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager > - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming > WordCount) changed to FAILED. > > > > java.lang.Exception: Call to registerInputOutput() of invokable > > failed > > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) > > > > at java.lang.Thread.run(Thread.java:745) > > > > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: > Cannot load user class: > org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper > > > > ClassLoader info: URL ClassLoader: > > > > Class not resolvable through given classloader. > > > > at > org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(St > reamConfig.java:219) > > > > at > org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutpu > t(StreamTask.java:135) > > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) > > > > ... 1 more > > > > > > > > I have no idea to fix this and need some help. > > > > > > > > Thanks! > > > > > > > > Greetings, > > Huang Wei > > 华为技术有限公司 Huawei Technologies Co., Ltd. > > > > > > Tel:+86 18106512602 > > Email:[hidden email] > > > > |
Thanks, changing the numerOfTaskSlots did it.
On Thu, 20 Aug 2015 at 15:08 huangwei (G) <[hidden email]> wrote: > Hi Aljoscha, > > Once, I had the problem like yours. > What I did is to set the taskmanager.numberOfTaskSlots: 4 in > conf/flink-conf.yaml since the parallelism of > StormWordCountRemoteBySubmitter is 4 by default. > Maybe you can take a try. > > BTW: You are right. The StormWordCountLocal gets access in IDEA. > > Hi Matthias, > Thank you for the help. > I`ll try. > > =========================== > > Hi, > I'm afraid submitting the correct program also doesn't work right now. > When I try to execute it I get this: > bin/flink run --jarfile > > /Users/aljoscha/Dev/work/flink/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/target/flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar > -c > > org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter > /Users/aljoscha/Downloads/storm-wc /Users/aljoscha/Downloads/storm-out > org.apache.flink.client.program.ProgramInvocationException: The main > method caused an error. > at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452) > at > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) > at org.apache.flink.client.program.Client.run(Client.java:327) > at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:608) > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:927) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > Caused by: NotAliveException(msg:null) > at > > org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:196) > at > > org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:190) > at > > org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) > ... 6 more > > The exception above occurred while trying to run your command. > > and no output is produced. Hopefully I'm doing something wrong. Any ideas > Matthias? > > By the way, the StormWordCountLocal works when executing in the IDE. > > Greetings, > Aljoscha > > On Thu, 20 Aug 2015 at 13:47 Matthias J. Sax < > [hidden email]> > wrote: > > > Hi Huang, > > > > you are using the wrong jar file. The current built, does not assemble > > a jar file for StormWordCountLocal. You can extend pom.xml. Look at > > the example StormWordCountRemoteBySubmitter and the corresponding > > assembly file word-count-storm.xml. > > > > What you need to do is: > > - add a new <plugin> tag similar to StormWordCountRemoteBySubmitter > > -> change descriptor file, program-class, and id > > - copy word-count-storm.xml to new descriptor file and adjust it > > accordingly (needs to include StormWordCountLocal instead of > > StormWordCountRemoteBySubmitter (and maybe other adjustments) > > - run maven -DskipTests package to build the new jar file > > > > If you have further problems, just let us know. > > > > -Matthias > > > > > > On 08/20/2015 01:19 PM, huangwei (G) wrote: > > > Hi, > > > > > > I got some new problems about the storm compatibility currently. > > > These occurred when I ran the “storm-wordcount” in the storm > > compatibility on a flink-0.10SNAPSHOT which I built it over a latest > > flink project. > > > > > > > > > First, I start a local flink: > > > > > > $ cd bin > > > > > > $ ./start-local.sh > > > > > > > > > > > > Then I ran the example: > > > > > > $ bin/flink run --jarfile > > ./examples/flink-storm-compatibility-examples-0.10-SNAPSHOT.jar -c > > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal > > > > > > > > > > > > It took a trouble like following: > > > > > > Executing StormWordCount* example with built-in default data > > > > > > Provide parameters to read input data from a file > > > > > > Usage: StormWordCount* <text path> <result path> > > > > > > java.lang.ExceptionInInitializerError > > > > > > at > > backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder > > .java:106) > > > > > > at > > org.apache.flink.stormcompatibility.api.FlinkTopologyBuilder.createTop > > ology(FlinkTopologyBuilder.java:78) > > > > > > at > > org.apache.flink.stormcompatibility.wordcount.StormWordCountLocal.main > > (StormWordCountLocal.java:66) > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > > ava:62) > > > > > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > > orImpl.java:43) > > > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > > > at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(Package > > dProgram.java:437) > > > > > > at > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeF > > orExecution(PackagedProgram.java:353) > > > > > > at org.apache.flink.client.program.Client.run(Client.java:327) > > > > > > at > > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:60 > > 8) > > > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:296) > > > > > > at > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:9 > > 27) > > > > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:977) > > > > > > Caused by: java.lang.RuntimeException: Could not find config file on > > classpath defaults.yaml > > > > > > at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:129) > > > > > > at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:160) > > > > > > at backtype.storm.utils.Utils.readStormConfig(Utils.java:184) > > > > > > at backtype.storm.utils.Utils.<clinit>(Utils.java:71) > > > > > > ... 14 more > > > > > > > > > > > > And then I added the default.yaml into > > flink-storm-compatibility-examples-0.10-SNAPSHOT.jar. > > > > > > It took another issue in the client log: > > > > > > 18:55:42,526 INFO org.apache.flink.runtime.jobmanager.JobManager > > - Status of job cb8ec2988613290d31eb6496c0af2e29 (Streaming > > WordCount) changed to FAILED. > > > > > > java.lang.Exception: Call to registerInputOutput() of invokable > > > failed > > > > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:526) > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > Caused by: > org.apache.flink.streaming.runtime.tasks.StreamTaskException: > > Cannot load user class: > > org.apache.flink.stormcompatibility.wrappers.StormBoltWrapper > > > > > > ClassLoader info: URL ClassLoader: > > > > > > Class not resolvable through given classloader. > > > > > > at > > org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(St > > reamConfig.java:219) > > > > > > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutpu > > t(StreamTask.java:135) > > > > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:523) > > > > > > ... 1 more > > > > > > > > > > > > I have no idea to fix this and need some help. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Greetings, > > > Huang Wei > > > 华为技术有限公司 Huawei Technologies Co., Ltd. > > > > > > > > > Tel:+86 18106512602 > > > Email:[hidden email] > > > > > > > > |
Free forum by Nabble | Edit this page |