Hi XiangWei,
Thank you for the inputs. I agree with you that it is possible that
containers may use extra memory in 1.8. As for native memory, it is memory
used by JVM and other processes outside JVM. So it's not limited by
MaxDirectMemorySize.
The community is working on a refactoring plan about resource management of
Flink. AFAIK, the native memory is on the table, but I'm not sure whether
the metaspace memory is considered. I think we should create a jira issue
on this.
Thank you~
Xintong Song
On Mon, May 20, 2019 at 4:47 PM XiangWei Huang <
[hidden email]>
wrote:
> Hi all,
> Currently i am running my flink application in yarn session mode and
> using below commnad :
> *bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096*
> when taskmanager complete to started,i found the container launching
> command is :
> * bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ......*
> from flink source code,i'v learned how direct memory size and heap memory
> size calculated :
> container total memory: 4096M
> network memory fraction: 0.1
> cut off memory fraction: 0.25
> networkMemory = 4096*0.1 = 409.6
> cut off memory = (4096 - 409.6) * 0.25 = 921.6
> *directMemorySize* = 409.6 + 921.6 = 1331.2
> *heapsize *= 4096 - 1331.2 = 2764.8
>
> Below is the environment i'm using to run yarn and flink:
> * jdk version: 1.8*
> * flink version: 1.6.1*
> *OS: centos7*
>
> there are two questions about memory allocation i want to ask:
> 1. Since the jdk version i an using is 1.8,is it necessary to consider the *metaspace
> memory *into calculation.According to the current way to calculate memory
> size（without metaspace memory） may cause
> container running beyond physical memory limit.
> 2. Is the native memory that rocksdb using part of direct memory（limit by
> jvm parameter *MaxDirectMemorySize*）,if not how do i control the size it
> used.
>
Locked
