Posted by
Shang Yuanchun (Jira) on
Jun 13, 2014; 7:45pm
URL: http://deprecated-apache-flink-mailing-list-archive.368.s1.nabble.com/jira-Commented-FLINK-934-Job-and-TaskManager-startup-scripts-always-set-JVM-heap-size-tp232.html
[
https://issues.apache.org/jira/browse/FLINK-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031071#comment-14031071 ]
ASF GitHub Bot commented on FLINK-934:
--------------------------------------
GitHub user rmetzger opened a pull request:
https://github.com/apache/incubator-flink/pull/18 Do not limit the client's heap space (See FLINK-934)
Similar to FLINK-934 and
https://github.com/apache/incubator-flink/pull/14 we should not limit the heap space of the job submission client to 512MB.
A user actually had the following exception while using Flink:
```
java.lang.OutOfMemoryError: Java heap space
at eu.stratosphere.runtime.io.serialization.DataOutputSerializer.resize(DataOutputSerializer.java:243)
at eu.stratosphere.runtime.io.serialization.DataOutputSerializer.write(DataOutputSerializer.java:87)
at eu.stratosphere.nephele.jobgraph.JobGraph.writeRequiredJarFiles(JobGraph.java:706)
at eu.stratosphere.nephele.jobgraph.JobGraph.write(JobGraph.java:641)
at eu.stratosphere.nephele.ipc.RPC$Invocation.write(RPC.java:153)
at eu.stratosphere.nephele.ipc.Client$Connection.sendParam(Client.java:469)
at eu.stratosphere.nephele.ipc.Client.call(Client.java:689)
at eu.stratosphere.nephele.ipc.RPC$Invoker.invoke(RPC.java:250)
at com.sun.proxy.$Proxy1.submitJob(Unknown Source)
at eu.stratosphere.nephele.client.JobClient.submitJobAndWait(JobClient.java:258)
at eu.stratosphere.client.program.Client.run(Client.java:284)
at eu.stratosphere.client.program.Client.run(Client.java:268)
at eu.stratosphere.client.program.Client.run(Client.java:262)
at eu.stratosphere.client.program.ContextEnvironment.execute(ContextEnvironment.java:50)
at eu.stratosphere.example.java.wordcount.WordCount.main(WordCount.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at eu.stratosphere.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:384)
at eu.stratosphere.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:302)
at eu.stratosphere.client.program.Client.run(Client.java:220)
at eu.stratosphere.client.CliFrontend.executeProgram(CliFrontend.java:327)
at eu.stratosphere.client.CliFrontend.run(CliFrontend.java:314)
at eu.stratosphere.client.CliFrontend.parseParameters(CliFrontend.java:927)
at eu.stratosphere.client.CliFrontend.main(CliFrontend.java:951)
```
I verified that the change resolves the problem.
You can merge this pull request into a Git repository by running:
$ git pull
https://github.com/rmetzger/incubator-flink stratosphere_client_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/18.patchTo close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18
----
commit 8e1bff41a21d75f5e7dc78fbc5c30835a3802781
Author: Robert Metzger <
[hidden email]>
Date: 2014-06-13T19:40:34Z
Do not limit the client's heap space (See FLINK-934) [ci skip]
----
> Job and TaskManager startup scripts always set JVM heap size
> ------------------------------------------------------------
>
> Key: FLINK-934
> URL:
https://issues.apache.org/jira/browse/FLINK-934> Project: Flink
> Issue Type: Improvement
> Reporter: Ufuk Celebi
> Assignee: Ufuk Celebi
> Fix For: 0.6-incubating
>
>
> The default distribution config {{stratosphere-conf.yaml}} contains keys {{jobmanager.heap.mb}} and {{taskmanager.heap.mb}} for the heap sizes of the JobManager and TaskManager JVMs. If they are not set, {{config.sh}} has default values for them.
> We _always_ use one of these values and set the {{Xms}} and {{Xmx}} properties. After a short discussion with [~rmetzger] and [~StephanEwen] we actually expected that they should not be set if not found in {{stratosphere-conf.yaml}}. This would result in a fall back to the respective JVM default (imho 1/4th of the physical memory since Java 6).
--
This message was sent by Atlassian JIRA
(v6.2#6252)