[
https://issues.apache.org/jira/browse/FLINK-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034295#comment-14034295 ]
Stephan Ewen commented on FLINK-939:
------------------------------------
Daniel, great to hear it!
I think a first prototype can make the "in memory" assumption. We may need to go to a "blob on disk" solution fairly soon. We are already from a user with a JobManager (running with 512 mbytes in a yarn container) that receives a 90 mb jar file may hit a limit (with heap fragmentation, two copies of the array due to resizing, and other data structures occupying space). A "blob" on disk that uses something like a weak hash map to cache in memory might be a cool solution for the future.
Concerning the integration with RPC: I think the current one does not urgently need that right now (having enough memory solves the issue for the time being), so it may not be worth to invest time there. I am not 100& sure how far the akka branch is, so I would suggest to build it standalone in the current branch and then we can make use of it when we merge the akka branch.
> Distribute required JAR files with seperate service
> ---------------------------------------------------
>
> Key: FLINK-939
> URL:
https://issues.apache.org/jira/browse/FLINK-939> Project: Flink
> Issue Type: Improvement
> Reporter: Ufuk Celebi
> Assignee: Daniel Warneke
>
> Currently, required user JAR files are distributed via the RPC service in {{JobGraph.writeRequiredJarFiles(DataOutput, AbstractJobVertex[])}}. The RPC service then tries to allocate a buffer on the client side heap to write the on-disk JAR, which [can lead to problems|
https://github.com/apache/incubator-flink/pull/18].
> This should be replaced with a seperate service.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Locked
